Hardware Requirements
Hardware requirements for server components.
Transfer Server
The transfer server is the central component on the part of the universities and research institutions. This is where the ingestor is installed and the metadata extractors are stored.
| Component | Minimum Requirements | Recommended Requirements |
|---|---|---|
| Memory | 8 GB | 16 GB or more |
| CPU | 4 cores | 8 cores or more |
| Network | 1 Gbps | 10 Gbps or more |
| Local Storage | 80 GB | 120 GB SSD or more |
The transfer server must be capable of transferring large amounts of data (approximately 1-2 TB) using Globus or S3 to another server. Additionally, the server will run metadata extractors that analyze hundreds of small text files and extract their contents. Smaller sites may be able to use a virtual machine for this role, but sites with large data transfer requirements will generally want a dedicated machine for good performance.
Most sites will run both the ingestor and Globus from the same machine. It is possible to run them from separate machines, but both systems should share data storage.
Cache Storage
OpenEM is flexible and can adapt to existing data storage systems. The most common are either to use microscope storage directly (by mounting storage from each detector on systems running OpenEM services) or to have a large central storage system that data gets copied to after acquisition. Either way, you should be clear on how data is organized and which users should have access to the data.
The cache storage should provide enough capacity to hold datasets until they can be archived; typically a minimum of 30 days is recommended.
| Component | Minimum Requirements | Recommended Requirements |
|---|---|---|
| Storage | 50 TB | 100 TB or more |
Software Requirements
Software requirements for server components
Operating System
Linux is required for Globus Connect Server.
The ingestor software can run on either Linux or Windows.
The software components can be compiled for different target operating systems. It is important that the operating systems are kept up to date and regularly supplied with updates.
Additional Packages
The OpenEM Standard Deployment uses docker compose to run the ingestor service. Running without docker is possible but requires the binaries and configuration to be manually upgraded.
Network Requirements
Network Traffic overview. Bold lines indicate data movement, while thin lines show HTTPS calls.
Firewall rules
Network isolation is important to the security of EM facilities. OpenEM has been designed not to require any ports to be accessible from the general internet. However, firewalls must be configured to allow traffic to and from some trusted hosts.
The following ports should be open for a standard configuration using globus to transfer data to PSI from the same machine that runs the ingestor.
| Service | Port | Source | Destination | Reason |
|---|---|---|---|---|
| Globus | tcp/443 | ingestor-server | 54.237.254.192/29 | Globus Control Out |
| Globus | tcp/443 | 54.237.254.192/29 | ingestor-server | Globus Control In |
| Globus | tcp/50000-51000 | ingestor-server | 192.33.126.53 (lx-globus-01.psi.ch) 192.33.126.54 (lx-globus-02.psi.ch) |
Globus GridFTP Out |
| Ingestor | tcp/4431 | User workstations | ingestor-server | Ingestor API |
| SciCat | tcp/443 | User workstations | discovery.psi.ch discovery-qa.psi.ch2 discovery.development.psi.ch2 |
SciCat frontend |
| SciCat | tcp/443 | ingestor-server User workstations |
dacat.psi.ch dacat-qa.psi.ch2 scicat.development.psi.ch2 |
SciCat backend |
| SciCat | tcp/443 | ingestor-server User workstations |
globus-proxy.psi.ch globus-proxy.development.psi.ch2 |
OpenEM globus proxy |
Domain names
All OpenEM traffic is encrypted with HTTPS. Modern browsers will refuse session sharing and cross-origin resource sharing without valid HTTPS certificates.
Each facility should register two domains:
- Globus (
em-globus.facility.chin examples) - Ingestor (
em-ingestor.facility.chin examples)
Usually both domains should be added to your DNS server as CNAME records resolving to
the actual hostname of the transfer server. A reverse proxy is then used to direct
traffic to the correct service (see installation chapter for
details).
See facility overview for current facility domain names.