Managing Environments with Conda and pip

A Python environment is more than a directory of importable packages. It also includes the Python interpreter and, in many scientific projects, compiled libraries that live outside the Python ecosystem. Managing these pieces together is where Conda is particularly useful.

pip and Conda overlap, but they solve different problems. pip installs Python distributions, usually from PyPI. Conda manages complete environments and can install Python, Python packages, native libraries, and command-line programs from Conda channels. I use Conda as the owner of the environment and pip as a fallback when a dependency is only available from PyPI.

Conda and pip logo

This distinction is especially valuable in data science and cheminformatics, where packages such as RDKit depend on compiled code. A plain virtual environment is often sufficient for a pure-Python application; Conda becomes attractive when the interpreter and non-Python dependencies must be reproducible as well.

Create a focused environment

Both Anaconda and Miniconda provide the conda command. Anaconda includes a large collection of packages, while Miniconda starts with a smaller installation and lets us add only what a project needs.

For a small data-analysis environment, we can request Python 3.8, pandas, Matplotlib, and JupyterLab in one transaction, with conda-forge as the highest-priority channel:

$ conda create --name data-workbench \
    --channel conda-forge \
    python=3.8 pandas matplotlib jupyterlab

The environment name identifies an isolated prefix, while python=3.8 is a version constraint that Conda’s solver must satisfy together with every package dependency. Installing the initial stack in one command lets the solver consider the complete request instead of resolving each package against a gradually changing environment.

Activate the environment before running its programs or installing more dependencies:

$ conda activate data-workbench

The active environment contributes its own Python executable and scripts to the shell. We can verify that boundary instead of relying only on the prompt:

$ python --version
Python 3.8.8

$ conda info --envs
# conda environments:
#
base                     /home/niklas/anaconda3
data-workbench        *  /home/niklas/anaconda3/envs/data-workbench

The asterisk marks the active environment. conda list shows the packages installed inside it:

$ conda list

Treat channels as part of the dependency model

A Conda channel is a package repository. The package name alone is therefore not the complete input: channel order also influences which build the solver selects.

For a one-off operation, --channel keeps that choice visible in the command. We can also search a specific channel before installing a package:

$ conda search --channel conda-forge rdkit

If conda-forge is the regular source for several projects, it can be added to the Conda configuration:

$ conda config --add channels conda-forge
$ conda config --set channel_priority strict

This produces a ~/.condarc configuration similar to:

channels:
  - conda-forge
  - defaults
channel_priority: strict

With strict priority, Conda prefers packages from the higher-priority channel instead of freely mixing builds with the same name across channels. This reduces the solver’s search space and makes the selected package source easier to reason about. The tradeoff is that a lower-priority build will not be considered when that package name exists in a higher-priority channel.

Manage environments without losing track of them

Conda can keep many environments side by side. List them with:

$ conda env list

Cloning is useful before an experiment or a risky dependency change:

$ conda create --name data-workbench-copy \
    --clone data-workbench

The clone is a snapshot for local experimentation, not a replacement for a checked-in environment definition. It preserves the installed state on this machine but does not explain which dependencies the project actually intends to use.

When the experiment is finished, leave and remove the cloned environment:

$ conda deactivate
$ conda env remove --name data-workbench-copy

Conda also keeps downloaded package archives in a shared cache so that several environments can reuse them. Removing cached archives can recover disk space without deleting an environment:

$ conda clean --tarballs

For broader cleanup options, inspect conda clean --help first. A cache is an optimization, so deleting all of it trades disk space for future downloads and package extraction.

Describe the environment as data

A sequence of successful shell commands is not yet a reproducible setup. The project should record its environment in a file that can be reviewed, versioned, and recreated:

name: data-workbench
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - pandas
  - matplotlib
  - jupyterlab
  - pip

Save this as environment.yml and create the environment with:

$ conda env create --file environment.yml

This file expresses the project’s direct requirements and channel order. Conda resolves compatible transitive dependencies for the target platform.

Conda can also generate environment files. A complete export records the resolved environment, including transitive packages and build details:

$ conda env export --name data-workbench > environment-full.yml

That is useful as a detailed snapshot, particularly when recreating an environment on the same platform. It can be unnecessarily restrictive across operating systems because platform-specific packages and build identifiers may be included.

For a smaller file based on the packages that were explicitly requested, export from the environment’s history:

$ conda env export --name data-workbench \
    --from-history > environment.yml

The two exports answer different questions: the full export describes what is installed, while the history-based export describes what Conda was asked to install. For a shared project, I prefer to review the smaller file and keep only the constraints that are part of the project’s real compatibility contract.

Use pip as a deliberate fallback

Some Python packages are published only on PyPI. They can still be installed inside an active Conda environment, but the order matters. Install the Conda dependencies first, make sure pip belongs to the environment, and use pip only for the remaining packages. Replace the placeholder below with the real package name:

$ conda install pip
$ python -m pip install your-pypi-only-package

Using python -m pip makes the interpreter-package relationship explicit: pip installs into the environment owned by the active Python executable. Avoid --user, because a user-level installation weakens the environment boundary and can shadow packages managed by Conda.

Once pip has changed the environment, running more Conda transactions may replace dependencies without considering every file installed by pip. A predictable workflow is therefore:

Install as much as possible with Conda.
Install the remaining PyPI packages with pip.
If the Conda requirements change, update the definition and recreate the environment.

Pip dependencies can live directly in environment.yml:

name: data-workbench
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - pandas
  - matplotlib
  - jupyterlab
  - pip
  - pip:
      - your-pypi-only-package==1.2.3

For a pip-only project, the familiar requirements workflow remains useful:

$ python -m pip freeze > requirements.txt
$ python -m pip install --requirement requirements.txt

pip freeze reports the installed Python distributions in requirements-file format. It is a snapshot, not a description of direct dependencies and not a record of the Python interpreter or native libraries. That is why requirements.txt alone cannot reproduce a complete Conda environment.

Choose the smallest tool that owns the whole problem

Conda is not automatically the right choice for every Python project. A pure-Python application may need only venv and pip. For scientific applications with compiled dependencies, several Python versions, or tools outside PyPI, Conda can make the complete environment easier to create and share.

The important rules are simple: make channel order explicit, store the environment definition with the project, install Conda packages before pip packages, and prefer recreating an environment over repairing an unknown state. The result is not only an isolated workspace but a setup another engineer can understand.

To see the other side of the ecosystem—publishing a Python project as a Conda package—continue with Publishing a Pure-Python CLI with Conda.