CVE-2023-35845: Anaconda3 creates numerous world-writable files on install

Background

Well over a year ago I installed Anaconda3 on my laptop for some work.  When the installation completed I noticed that numerous files had been installed world-writable.  This is less then ideal obviously but it was my laptop so I fixed the issue and moved on.  Fast-forward to September 2022 and I'm left cleaning up these issues after every upgrade or install at a customer site.  This is where I become annoyed that this issue still hasn't been resolved so I figured I better report it.

After an unacceptably long delay to try to get resolution of this issue I am publishing the details.

 

Technical Details

Installing any version of the Anaconda or Miniconda on Linux results in numerous world-writable files.  For example, below I took the latest version from the website and performed the batch (no prompt) install to $HOME/anaconda3:

 

[jeremy@test1 ~]$ bash Anaconda3-2023.03-1-Linux-x86_64.sh -b -p $HOME/anaconda3
PREFIX=/home/jeremy/anaconda3
Unpacking payload ...

Installing base environment...


Downloading and Extracting Packages


Downloading and Extracting Packages

Preparing transaction: done
Executing transaction: |

    Installed package of scikit-learn can be accelerated using scikit-learn-intelex.
    More details are available here: https://intel.github.io/scikit-learn-intelex

    For example:

        $ conda install scikit-learn-intelex
        $ python -m sklearnex my_application.py



done
installation finished.

With the installation finished I ran a quick check of the permissions for files and directories with the other writable bit set.  As you can see from the two commands below they are all files.   Previously I have found directories as well.

[jeremy@test1 ~]$ find /home/jeremy/anaconda3 -perm /0002 \( -type f -o -type d \) | wc
    606     606   58235
[jeremy@test1 ~]$ find /home/jeremy/anaconda3 -perm /0002 -type f | wc
    606     606   58235
 

To see what those files look like permission wise you can run a find command filtering for files that are world-writable with the "-perm /0002" option:

 

[jeremy@test1 ~]$ find /home/jeremy/anaconda3 -perm /0002 -type f -ls
  3773199     92 -rwxrwxrwx   2  jeremy   jeremy      91648 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/distlib/w32.exe
  3773231    108 -rwxrwxrwx   2  jeremy   jeremy     108032 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/distlib/t64.exe
  3773242    168 -rwxrwxrwx   2  jeremy   jeremy     168448 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/distlib/w64-arm.exe
  3773208     96 -rwxrwxrwx   2  jeremy   jeremy      97792 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/distlib/t32.exe
  3773215    100 -rwxrwxrwx   2  jeremy   jeremy     101888 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/distlib/w64.exe
  3773253    180 -rwxrwxrwx   2  jeremy   jeremy     182784 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/distlib/t64-arm.exe
  3773149      4 -rw-rw-rw-   2  jeremy   jeremy        469 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/vendor.txt
  3773176    280 -rw-rw-rw-   2  jeremy   jeremy     286370 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem
  3773161      4 -rw-rw-rw-   2  jeremy   jeremy        286 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/py.typed
  3773169      4 -rw-rw-rw-   2  jeremy   jeremy       4072 Dec  9  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip-22.3.1-py3.10.egg-info/PKG-INFO
...
  3713940      4 -rwxrwxrwx   1  jeremy   jeremy        740 Aug 31  2020 /home/jeremy/anaconda3/pkgs/libllvm10-10.0.1-hbcb73fb_5/info/recipe/parent/install_llvm.bat
  3714015      4 -rwxrwxrwx   1  jeremy   jeremy       1387 Aug 31  2020 /home/jeremy/anaconda3/pkgs/libllvm10-10.0.1-hbcb73fb_5/info/recipe/parent/install_llvm.sh
  3713958      4 -rwxrwxrwx   1  jeremy   jeremy        726 Sep  4  2020 /home/jeremy/anaconda3/pkgs/libllvm10-10.0.1-hbcb73fb_5/info/recipe/parent/xcode-select
  3714009      4 -rwxrwxrwx   1  jeremy   jeremy       1387 Aug 31  2020 /home/jeremy/anaconda3/pkgs/libllvm10-10.0.1-hbcb73fb_5/info/recipe/install_llvm.sh
  3657118      4 -rwxrwxrwx   1  jeremy   jeremy       3364 Jan  5  2021 /home/jeremy/anaconda3/pkgs/jupyterlab_widgets-1.0.0-pyhd3eb1b0_1/info/recipe/recipe_log.txt
  3657112      4 -rwxrwxrwx   1  jeremy   jeremy        828 Jan  5  2021 /home/jeremy/anaconda3/pkgs/jupyterlab_widgets-1.0.0-pyhd3eb1b0_1/info/recipe/meta.yaml.template
  3279942      4 -rwxrwxrwx   1  jeremy   jeremy       1923 Jun 13  2019 /home/jeremy/anaconda3/pkgs/backports.weakref-1.0.post1-py_1/info/recipe/recipe_log.txt
  3279943      4 -rwxrwxrwx   1  jeremy   jeremy       1091 Jun 12  2019 /home/jeremy/anaconda3/pkgs/backports.weakref-1.0.post1-py_1/info/recipe/meta.yaml.template
  3709519      8 -rwxrwxrwx   1  jeremy   jeremy       4402 May  4  2020 /home/jeremy/anaconda3/pkgs/atomicwrites-1.4.0-py_0/info/recipe/recipe_log.txt
  3709528      4 -rwxrwxrwx   1  jeremy   jeremy        774 May  4  2020 /home/jeremy/anaconda3/pkgs/atomicwrites-1.4.0-py_0/info/recipe/meta.yaml.template
 

I'm not sure how frequently if ever these files are used but one file stuck out to me when I reviewed the list.  The cacert.pem file used by the python certify module under their pip installation.  Since certify provides the trusted root certificate authorities, any manipulation of these could allow untrusted content to pass through SSL connections via Man-in-the-middle attack.  You can see below there are two files found but since they are the same inode number we know they are actually a hardlink:

[jeremy@test1 ~]$ find /home/jeremy/anaconda3 -type f -perm /0002 -name cacert.pem -ls
  1986492    280 -rw-rw-rw-   2  jeremy   jeremy     286370 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem
  1986492    280 -rw-rw-rw-   2  jeremy   jeremy     286370 Nov  5  2022 /home/jeremy/anaconda3/pkgs/pip-22.3.1-py310h06a4308_0/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem

For illustrations purposes here I created and used a different account (test) to truncate the cacert.pem files and show they have been truncated to 0 bytes.

[test@test1 ~]$ find /home/jeremy/anaconda3/ -type f -perm /0002 -name cacert.pem -ls
  1986492    280 -rw-rw-rw-   2  jeremy   jeremy     286370 Nov  5  2022 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem
  1986492    280 -rw-rw-rw-   2  jeremy   jeremy     286370 Nov  5  2022 /home/jeremy/anaconda3/pkgs/pip-22.3.1-py310h06a4308_0/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem
[test@test1 ~]$ truncate --size=0 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem
[test@test1 ~]$ find /home/jeremy/anaconda3/ -type f -perm /0002 -name cacert.pem -ls
  1986492      0 -rw-rw-rw-   2  jeremy   jeremy          0 Jun  5 17:28 /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem
  1986492      0 -rw-rw-rw-   2  jeremy   jeremy          0 Jun  5 17:28 /home/jeremy/anaconda3/pkgs/pip-22.3.1-py310h06a4308_0/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem

Moving back to my normal user,  if these files are used by the Anaconda3 packaged version of pip this should result in SSL error:

[jeremy@test1 ~]$ /home/jeremy/anaconda3/bin/pip install tensorflow==
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4123)'))': /simple/tensorflow/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4123)'))': /simple/tensorflow/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4123)'))': /simple/tensorflow/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4123)'))': /simple/tensorflow/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4123)'))': /simple/tensorflow/
Could not fetch URL https://pypi.org/simple/tensorflow/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/tensorflow/ (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4123)'))) - skipping
ERROR: Could not find a version that satisfies the requirement tensorflow== (from versions: none)
ERROR: No matching distribution found for tensorflow==

As you can see we do in fact get an error that no valid cert is found.  This is due to the truncated root CA certs file.  Clearly there are significant risks besides a Denial of Service (DoS) from Anaconda's creation of these world-writable files. 

Disclosure Timeline:

• Sep 22, 2022 - Reported world-writable vulnerability including list of files
  
• Sep 22, 2022 - Confirmation from Anaconda
• Sep 23, 2022 - Provided an easy way to reproduce to Anaconda
• Sep 23, 2022 - Anaconda confirms reproducer and says "We are working on a fix for this issue and will notify you. once a fix is available"
• Nov 16, 2022 - I request status update
• Nov 16, 2022 - Anaconda responds "Our security team is taking a look at this issue this week and we can give you a better time frame estimation once this has been completed. "
• Nov 23, 2022 - Anaconda updates "Our security group has reviewed this issue, and are prioritizing it for a future sprint cycle"
• Jan 31, 2023 - With no response from Anaconda I test and the issue still exists.  At this point I research if a CVE is filed and find similar (though not the sames) issues in Windows reported as: CVE-2022-26526.
• Feb 20, 2023 - Anaconda responds that CVE-2022-26526 is fixed but the issue I reported "is unfortunately not a straightforward fix".
• Feb 28, 2023 - I respond that any proper fix should take into account the user's umask during installation.
• Feb 28, 2023 - Anaconda acknowledges my umask email
• Jun 1, 2023 - I inquire again for the status of a fix
• Jun 1, 2023 - Anaconda erroneously believes "fixes for this implemented in Anaconda 2022.05 and Miniconda3 4.12.0 and beyond"
• Jun 2, 2023 - I verify Linux is still vulnerable and email them to let them know
• Jun 2, 2023 - I finally file for a CVE with MITRE as the CNA-LR
• Jun 18, 2023 - MITRE rerves CVE-2023-35845 for this issue
• Jun 22, 2023 - I provide one more email to Anaconda notifying them of the impending CVE and receive no response as of this writing on Jun 27.
  

Solutions Given this information it's clear that there are risks and possibly significant risks besides a Denial of Service (DoS) from Anaconda's creation of these world-writable files.  Luckily if you already  upgraded the pip packaged with Anaconda it appears to fix this world-writable cacert.pem file limiting the one illustrated path to exploitation here: [jeremy@test1 ~]$ $HOME/anaconda3/bin/pip install --upgrade pip Requirement already satisfied: pip in ./anaconda3/lib/python3.10/site-packages (22.3.1) Collecting pip   Using cached pip-23.1.2-py3-none-any.whl (2.1 MB) Installing collected packages: pip   Attempting uninstall: pip     Found existing installation: pip 22.3.1     Uninstalling pip-22.3.1:       Successfully uninstalled pip-22.3.1 Successfully installed pip-23.1.2 You can see there does exist one of the cacert.pem file but this is not the one used by the Anaconda3 pip (a utility detailed below confirmed that for me): [jeremy@test1 ~]$ find $HOME/anaconda3 -type f -perm /0002 -name cacert.pem -ls   3773176    280 -rw-rw-rw-   1  jeremy   jeremy     286370 Nov  5  2022 /home/jeremy/anaconda3/pkgs/pip-22.3.1-py310h06a4308_0/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem   Regardless of the pip solution the best strategy for now appears to be to strip all "other" write permissions from your Anaconda3 installation directory any time you install or update software: [jeremy@test1 ~]$ find $HOME/anaconda3 -type f -perm /0002 -exec chmod o-w {} \;

Detection of Similar Problems Finally, if your interested in a utility to capture information about what opens, executes, or changes permissions to create world-writable I've create an eBPF program to help identify that and stored in my github repo here: https://github.com/jfilizetti/ebpf-security-tools Here is an example capture of the cacert.pem described above: [root@test1 jeremy]# ./world_writable_monitor.py -f regular,directory | grep cacert.pem chmod      666     1000:1000    93877     conda.exe             0                                    -100       /home/jeremy/anaconda3/pkgs/pip-22.3.1-py310h06a4308_0/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem open       666     1000:1000    93910     pip                   0                                    -100       /home/jeremy/anaconda3/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pem

 

Modern NVIDIA GPU Transcoding Comparison

Modern NVIDIA GPU Transcoding Comparison

There is nothing quite like new hardware.  I'm always looking for ways to speed up my daily video processing so GPUs are always top of mind.  Unfortunately I've never seen the kind of numbers I want from NVIDIA to know whether or not it's worth buying the latest and greatest of their data center class cards for encoding/decoding/transcoding/etc.  When the marketing info comes out for a new card like the fairly new NVIDIA L4 you are left wondering how it stacks up against your particular challenge.  I don't have 8 - L4s or a need to handle 1040 simultaneous 720p30 streams.  What I do have is a dozen 4K streams that I need reduced to 1080p for longer term storage.  I'd like to know how well a new L4 or even A2 card works for my purposes compared to my older cards like a GTX 1660 Ti or ancient GTX 1080.

Now that I've dumped the money into those cards I was surprised to see that the performance was not all that different compared to my oldest card which is a Pascal generation GTX 1080.  At this point it was unclear if it was just the fact that the GTX 1080 had dual encoder chips or maybe because I was still running FFMPEG 4.3.2.  I finally decided to pull and build the latest version of FFMPEG which was 5.1.2 at that time.  Rerunning all tests it became clear to me that what I once thought was my "flagship" card for encoding was actually pretty outdated with the new software.  All of my cards performed much better.   The following graphic illustrates those numbers.  

 

Note: For all the data below the FFMPEG command is roughly:

ffmpeg -vsync vfr -hwaccel_device 0 -i in.mp4 -vf "scale_npp=iw/2:ih/2" -vcodec hevc_nvenc -b:v 2M -r 15 out.mp4

 

It's probably worth a little explanation of the workload.  The data is comprised of 496 video files that are 5 minutes in length.   The breakdown is approximately:

• 100 - 4K 30 FPS HEVC videos
• 300 - 4K 15 FPS HEVC videos
• 100 - 5 MP 20 FPS H264 videos

Outside of the gains of the Turing, Ampere, and Ada Lovelace generation cards it's noteworthy that performance didn't really change for the Pascal generation GTX 1080 that I once consider my "flagship" cards.

 

Getting the most with the new version of FFMPEG

With the L4 card having 2 encoders and 4 decoders I found it necessary to throw more than one simultaneous transcode to get the most performance.  It seemed that 2 would be optimal but I've found that when processing H264 video that the utilization of the card would drop as the nvenc/nvdec seem to be able to handle more where HEVC would saturate the nvenc/nvdec chip.  You can see the stats during the processing of the prior mentioned dataset using a single transcode at a time.  All stats are for utilization are from "nvidia-smi dmon -d 1".

 

The drop around 4000 seconds and 6000 seconds are the prior mentioned H264 datasets.  Looking at two simultaneous transcodes you can see the processing time is reduced but not in half.

Finally I ran a test with 4 simultaneous transcodes and found the timeline dropped to about half of the original.

Given the above "optimal" processing on the L4 I produced some histograms of the FPS that were achieved for all of the prior mentioned cards as well as the various runs on the L4.

It's clear with 2 and 4 in parallel transcodes on the L4 there is more variability and the average FPS is lower but the overall dataset processes faster. 

 

Conclusion

1. With NVIDIA chipsets is wise to keep upgrading as FFMPEG and the NVIDIA Video Codec SDK is always making performance improvements.  

2.  An optimal amount of simultaneous transcode processes depends on the codec so some trial and error is necessary.  

3.  Most nvenc/nvdec chips process roughly around the same rate.  I've always thought this would not be the case.  In the future I may try to dig deeper on this.

External Resources

You can always consult the matrix they have to know how many encoder and decoder chips are available per card at: https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new