Pangeo Cloud¶

Operated by 2i2c¶

Pangeo Cloud is operated by 2i2c. You can see the deployment configuration here: https://github.com/2i2c-org/infrastructure/tree/master/config/clusters/pangeo-hubs

Quickstart¶

Getting Support¶

We would like for Pangeo Cloud to one day be a full-featured, professional service. But it’s not yet! It is an experiment, run by scientists and hackers who devote their nights and weekends to this project. Professional level support (a service-level agreement, a 24-hour helpdesk) is non-existent.

With those caveats, support requests for Pangeo Cloud can be made on our Discourse forum

• https://discourse.pangeo.io/c/Cloud/Tech-support-forum-for-Pangeo-Cloud/17

We use this forum to track technical support requests for Pangeo Cloud. Someone will do their best to respond to you as quickly as possible given the circumstances. This is not a place for general support requests about scientific python. This forum is specific to issues related to Pangeo Cloud such as:

• login problems

• environment configuration

• reading / writing cloud data

• Dask Gateway on Pangeo Cloud

In making a support request, please provide the following information:

• Which cluster are you using (e.g. us-central1-b.gcp.pangeo.io)

• A concise summary of your problem

• If your problem involves code, please do your best to include a minimum reproducible example

Sign Up¶

Pangeo Cloud is available to researchers across the world engaged in data-intensive Earth and Environmental Science. Prospective users must apply using the following form:

• Pangeo Cloud Application

Access will be granted based on a review of the proposed research project and its technical suitability for Pangeo Cloud.

If your application is approved, you will receive an invitation to join a GitHub Team in the pangeo-data organization. You must accept this invitation to gain access to Pangeo Cloud resources.

Hubs¶

Pangeo Cloud currently comprises two different JupyterHubs:

• us-central1-b.gcp.pangeo.io: A hub in Google Cloud Platform.

• aws-uswest2.pangeo.io: A hub in AWS.

We recommend choosing a hub based on what data you want to access. E.g. if your data of interest live primarily in AWS, choose aws-uswest2. (See AWS Public Datasets and Google Cloud Datasets.)

Once your application is approved, you will be able to log in to the resources you requested.

Software Environment¶

Each hub contains a customized data-science software environment based off the docker images built in https://github.com/pangeo-data/pangeo-docker-images. These are the steps for adding a package to the environment:

1. Make a pull request to https://github.com/pangeo-data/pangeo-docker-images updating the environment.yml file for the pangeo-notebook docker image at https://github.com/pangeo-data/pangeo-docker-images/blob/master/pangeo-notebook/environment.yml.

2. When that pull request is merged, the Pangeo maintainers will push a tag with today’s date, triggering a release of new docker images.

3. A bot will submit a pull request to pangeo-cloud-federation updating the images (https://github.com/pangeo-data/pangeo-cloud-federation/pull/733). When this is merged, you can test your changes in the staging environment (at either https://staging.us-central1-b.gcp.pangeo.io or https://staging.aws-uswest2.pangeo.io)

4. If things look good, the Pangeo maintainers can trigger a deploy to production, making the package avaiable to everyone.

Users may use pip and conda to install new packages in their own environments, but this approach currently has some limitations noted below. To install into your personal environment, it’s crucial that you include the --user flag to pip. This installs the package in ~/.local/lib/PYTHON/site-packages/. Since it’s in your home directory, it will persist across jupyterlab sessions.

• Changes in the environment are not propagated to Dask ps (though see below for a way to include packages on the workers too).

• Installing additional packages with pip into an existing conda environment risks breaking the environment if doesn’t see packages installed by conda and vice versa. You shouldn’t install / update packages that are already in the environment.

Hardware Environment¶

Pangeo Cloud clusters offer different amounts of RAM and CPU to the user notebook upon login. Please choose the least resource-intensive option for the work you need to do. Larger virtual machines cost us more money.

Files and Data in the Cloud¶

Please see the 2i2c documentation on Files and Data in the Cloud.

Dask¶

Dask is an important component of Pangeo Cloud and can be used to help parallelize large calculations. All environments support the standard multi-threaded dask scheduler, and by default, zarr-backed cloud data datasets will open in Xarray as collections of Dask arrays.

from dask_gateway import GatewayCluster

cluster = GatewayCluster()
cluster.adapt(minimum=2, maximum=10)  # or cluster.scale(n) to a fixed size.
client = cluster.get_client()

cluster.close()

from dask_gateway import Gateway
gateway = Gateway()
options = gateway.cluster_options()

# set the options programatically, or through their HTML repr
options.worker_memory = 10  # 10 GB of memory per worker.

# Create a cluster with those options
cluster = gateway.new_cluster(options)
cluster.scale(...)
client = cluster.get_client()

>>> gateway = Gateway()
>>> gateway.list_clusters()
[ClusterReport<name=prod.c288c65c429049e788f41d8308823ca8, status=RUNNING>]

# connect to the cluster
cluster = g.connect(g.list_clusters()[0].name)
# shut it down
cluster.close()

>>> from dask_gateway import Gateway
>>> gateway = Gateway()
>>> options = gateway.cluster_options()

>>> options.environment = {"MY_VARIABLE": "1"}

>>> cluster = gateway.new_cluster(options)
>>> cluster.scale(1)
>>> client = cluster.get_client()

>>> def check():
...     import os
...     return os.environ["MY_VARIABLE"]

>>> client.run(check)
{'tls://10.36.248.180:33361': '1'}

>>> env = {
...     "DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING": False,
...     "DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES": 5,
... }
>>> options.environment = env
>>> cluster = gateway.new_cluster(options)

>>> from dask.distributed import PipInstall
>>> from dask_gateway import GatewayCluster
>>> cluster = GatewayCluster()  # create the cluster nomrally
>>> client = cluster.get_client()
>>> # Now create and register the plugin. We'll install 'bulwark'
>>> plugin = PipInstall(packages=['bulwark'])
>>> client.register_worker_plugin(plugin)

.. code-block:: python

>>> def check():
...     import bulwark
...     return bulwark.__version__
>>> cluster.scale(2)
>>> client.wait_for_workers(2)
>>> client.run(check)
{'tls://10.36.248.117:40785': '0.6.1', 'tls://10.37.142.70:43031': '0.6.1'}