This article will describe how to download an image from a (docker) container registry.

Manual Download of Container Images with wget and curl

Intro

Remember the good `'ol days when you could just download software by visiting a website and click "download"?

Even apt and yum repositories were just simple HTTP servers that you could just curl (or wget) from. Using the package manager was, of course, more secure and convenient -- but you could always just download packages manually, if you wanted.

But have you ever tried to curl an image from a container registry, such as docker? Well friends, I have tried. And I have the scars to prove it.

It was a remarkably complex process that took me weeks to figure-out. Lucky you, this article will break it down.

Examples

Specifically, we'll look at how to download files from two OCI registries.

1. Docker Hub
2. GitHub Packages

Terms

First, here's some terminology used by OCI

1. OCI - Open Container Initiative
2. blob - A "blob" in the OCI spec just means a file
3. manifest - A "manifest" in the OCI spec means a list of files

Prerequisites

This guide was written in 2024, and it uses the following software and versions:

1. debian 12 (bookworm)
2. curl 7.88.1
3. OCI Distribution Spec v1.1.0 (which, unintuitively, uses the '/v2/' endpoint)

Of course, you'll need 'curl' installed. And, to parse json, 'jq' too.

sudo apt-get install curl jq

What is OCI?

OCI stands for Open Container Initiative.

OCI was originally formed in June 2015 for Docker and CoreOS. Today it's a wider, general-purpose (and annoyingly complex) way that many projects host files (that are extremely non-trivial to download).

One does not simply download a file from an OCI-complianet container registry. You must:

1. Generate an authentication token for the API
2. Make an API call to the registry, requesting to download a JSON "Manifest"
3. Parse the JSON Manifest to figure out the hash of the file that you want
4. Determine the download URL from the hash
5. Download the file (which might actually be many distinct file "layers")

One does not simply download from a container registry

In order to figure out how to make an API call to the registry, you must first read (and understand) the OCI specs here.

• https://opencontainers.org/release-notices/overview/

OCI APIs

OCI maintains three distinct specifications:

1. image spec
2. runtime spec
3. distribution spec

OCI "Distribution Spec" API

To figure out how to download a file from a container registry, we're interested in the "distribution spec". At the time of writing, the latest "distribution spec" can be downloaded here:

• https://github.com/opencontainers/distribution-spec/releases/tag/v1.1.0
• https://github.com/opencontainers/distribution-spec/releases/download/v1.1.0/oci-distribution-spec-v1.1.0.pdf

The above PDF file defines a set of API endpoints that we can use to query, parse, and then figure out how to download a file from a container registry. The table from the above PDF is copied below:

ID	Method	API Endpoint	Success	Failure
end-1	GET	/v2/	200	404/401
end-2	GET / HEAD	/v2/<name>/blobs/<digest>	200	404
end-3	GET / HEAD	/v2/<name>/manifests/<reference>	200	404
end-4a	POST	/v2/<name>/blobs/uploads/	202	404
end-4b	POST	/v2/<name>/blobs/uploads/?digest=<digest>	201/202	404/400
end-5	PATCH	/v2/<name>/blobs/uploads/<reference>	202	404/416
end-6	PUT	/v2/<name>/blobs/uploads/<reference>?digest=<digest>	201	404/400
end-7	PUT	/v2/<name>/manifests/<reference>	201	404
end-8a	GET	/v2/<name>/tags/list	200	404
end-8b	GET	/v2/<name>/tags/list?n=<integer>&last=<integer>	200	404
end-9	DELETE	/v2/<name>/manifests/<reference>	202	404/400/405
end-10	DELETE	/v2/<name>/blobs/<digest>	202	404/405
end-11	POST	/v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name>	201	404
end-12a	GET	/v2/<name>/referrers/<digest>	200	404/400
end-12b	GET	/v2/<name>/referrers/<digest>?artifactType=<artifactType>	200	404/400
end-13	GET	/v2/<name>/blobs/uploads/<reference>	204	404

In OCI, files are (cryptically) called "blobs". In order to figure out the file that we want to download, we must first reference the list of files (called a "manifest").

The above table shows us how we can download a list of files (manifest) and then download the actual file (blob).

Examples

Let's look at how to download files from a couple different OCI registries:

1. Docker Hub
2. GitHub Packages

Docker Hub

To see the full example of downloading images from docker hub, click here

GitHub Packages

To see the full example of downloading files from GitHub Packages, click here.

Why?

I wrote this article because many, many folks have inquired about how to manually download files from OCI registries on the Internet, but their simple queries are usually returned with a barrage of useless counter-questions: why the heck would you want to do that!?!

The answer is varied.

Some people need to get files onto a restricted environment. Either their org doesn't grant them permission to install software on the machine, or the system has firewall-restricted internet access -- or doesn't have internet access at all.

3TOFU

Personally, the reason that I wanted to be able to download files from an OCI registry was for 3TOFU.

Verifying Unsigned Releases with 3TOFU

Unfortunaetly, most apps using OCI registries are extremely insecure. Docker, for example, will happily download malicious images. By default, it doesn't do any authenticity verifications on the payloads it downloaded. Even if you manually enable DCT, there's loads of pending issues with it.

Likewise, the macOS package manager brew has this same problem: it will happily download and install malicious code, because it doesn't use cryptography to verify the authenticity of anything that it downloads. This introduces watering hole vulnerabilities when developers use brew to install dependencies in their CI pipelines.

My solution to this? 3TOFU. And that requires me to be able to download the file (for verification) on three distinct linux VMs using curl or wget.

⚠ NOTE: 3TOFU is an approach to harm reduction.

It is not wise to download and run binaries or code whose authenticity you cannot verify using a cryptographic signature from a key stored offline. However, sometimes we cannot avoid it. If you're going to proceed with running untrusted code, then following a 3TOFU procedure may reduce your risk, but it's better to avoid running unauthenticated code if at all possible.

Registry (ab)use

Container registries were created in 2013 to provide a clever & complex solution to a problem: how to package and serve multiple versions of simplified sources to various consumers spanning multiple operating systems and architectures -- while also packaging them into small, discrete "layers".

However, if your project is just serving simple files, then the only thing gained by uploading them to a complex system like a container registry is headaches. Why do developers do this?

In the case of brew, their free hosing provider (JFrog's Bintray) shutdown in 2021. Brew was already hosting their code on GitHub, so I guess someone looked at "GitHub Packages" and figured it was a good (read: free) replacement.

Many developers using Container Registries don't need the complexity, but -- well -- they're just using it as a free place for their FOSS project to store some files, man.

What's the best way to monitor and log which processes are responsible for high system load throughout the day? Tools like top and htop only provide immediate values, but I'm looking for a solution that offers historical data to identify the main culprits over time.

@sysadmin

#sysadmin #linux #server

Discussion question: Are we to centralized? (I know Lemmy isn't unbiased)

So using qemu with hyper-V acceleration is something that is not well documented. Historically, you would setup HAXM but that has been discontinued and deprecated.

To use qemu on WIndows with hardware acceleration you first start by enabling Hyper-V if it isn't enabled already. Next, run qemu with the following additional option:

--accel whpx

In qtemu on Windows there is a GUI option to do this. I like qemu because it cleaner than pure Hyper-V and doesn't have the licensing issues that Virtualbox does. I also like that Linux guests have native support for virtual devices.

https://www.qemu.org/docs/master/system/qemu-manpage.html

For those who want a alternative to Adobe without using Edge

So we run VMware, and this morning I go and check a thing, and Firefox gives me an error.. connection insecure cert is invalid

No I don’t have the exact verbiage

But Edge and Chrome opened it just fine. Whisky Tango?

It was a rekeyed , and re installed the cert for an easy ish fix.

But I’m far more weirded out that FF slapped it down ; and the other two were like; Ja sure no problem…

??

Maybe should x post to c/firefox as well

Took me a few hours to figure this out, figured I'd pass it along. Forgive formatting, I'm on mobile.

How to Bypass Bitlocker for Crowdstrike BSoD

Only use this if the Bitlocker key is lost.

 From the Bitlocker screen, select Skip This Drive. A command prompt will appear.

Type bcdedit /set {default} safeboot network and press Enter.

Type Exit to exit the command prompt, then select Shut Down

Hardwire the device to the network

Login as an admin account

Navigate to C:\Windows\System32\Drivers\Crowdstrike and delete C:\windows\system32\drivers\crowdstrike\c-00000291-*.sys

Win+R to open the Run menu, then type msconfig and press Enter

Go to Boot

Uncheck the box for SafeBoot

You will receive a warning about Bitlocker. Proceed.

Click OK and you will be prompted to restart. Do so.

Have the user login

Test their access to files

cross-posted from: https://sh.itjust.works/post/22460079

Today I'm grateful I'm using Linux - Global IT issues caused by Crowdstrike update causes BSOD on Windows

This isn't a gloat post. In fact, I was completely oblivious to this massive outage until I tried to check my bank balance and it wouldn't log in.

Apparently Visa Paywave, banks, some TV networks, EFTPOS, etc. have gone down. Flights have had to be cancelled as some airlines systems have also gone down. Gas stations and public transport systems inoperable. As well as numerous Windows systems and Microsoft services affected. (At least according to one of my local MSMs.)

Seems insane to me that one company's messed up update could cause so much global disruption and so many systems gone down :/ This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

cross-posted from: https://lemmy.ml/post/18154572

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

It keeps showing Internal server error and has been for over 2 hours now. I tested from different locations (China, US, EU) and same errors everywhere. Yet I don't see any news or posts about this anywhere. I have an exchange server that needs installing today and I can't download the prerequisites to even launch the install. Sigh

Just a quick fyi for anyone using Intune to distribute firewall rules with the "Endpoint Protection > Windows Firewall" profile for the first time.
Any rules you set won't be visible in the wf.msc GUI or using PowerShell's Get-NetFirewallRule.

The only place you can see those rules is the registry under this key

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\SharedAccess\Parameters\FirewallPolicy\Mdm\FirewallRules

This post contains a canary message that's cryptographically signed by the official BusKill PGP release key

The BusKill project just published their Warrant Canary #008

For more information about BusKill canaries, see:

• https://buskill.in/canary

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Status: All good
Release: 2024-06-11
Period: 2024-06-01 to 2024-12-31
Expiry: 2025-01-31

Statements
==========

The BusKill Team who have digitally signed this file [1]
state the following:

1. The date of issue of this canary is June 11, 2024.

2. The current BusKill Signing Key (2020.07) is

   E0AF FF57 DC00 FBE0 5635  8761 4AE2 1E19 36CE 786A

3. We positively confirm, to the best of our knowledge, that the 
   integrity of our systems are sound: all our infrastructure is in our 
   control, we have not been compromised or suffered a data breach, we 
   have not disclosed any private keys, we have not introduced any 
   backdoors, and we have not been forced to modify our system to allow 
   access or information leakage to a third party in any way.

4. We plan to publish the next of these canary statements before the
   Expiry date listed above. Special note should be taken if no new
   canary is published by that time or if the list of statements changes
   without plausible explanation.

Special announcements
=====================

None.

Disclaimers and notes
=====================

This canary scheme is not infallible. Although signing the 
declaration makes it very difficult for a third party to produce 
arbitrary declarations, it does not prevent them from using force or 
other means, like blackmail or compromising the signers' laptops, to 
coerce us to produce false declarations.

The news feeds quoted below (Proof of freshness) serves to 
demonstrate that this canary could not have been created prior to the 
date stated. It shows that a series of canaries was not created in 
advance.

This declaration is merely a best effort and is provided without any 
guarantee or warranty. It is not legally binding in any way to 
anybody. None of the signers should be ever held legally responsible 
for any of the statements made here.

Proof of freshness
==================

04 Jun 24 14:10:16 UTC

Source: DER SPIEGEL - International (https://www.spiegel.de/international/index.rss)
Fortress Europe: Migrants Abandoned on the Edge of the Sahara
Israel-Gaza-Krieg: Menschenrechtler Aryeh Neier über Schuldfrage und Strafverfolgung (Kopie)

Source: NYT > World News (https://rss.nytimes.com/services/xml/rss/nyt/World.xml)
Middle East Crisis: Israeli Airstrikes Kill Iranian General in Syria
Live Updates: India’s Election Results Suggest a Setback for Modi

Source: BBC News - World (https://feeds.bbci.co.uk/news/world/rss.xml)
Shock for India's Modi as opposition set to slash majority
Gaza ceasefire plan turns into deadly game of survival

Source: Bitcoin Blockchain (https://blockchain.info/q/latesthash)
000000000000000000014cd79802b29c1dcd7fc6debee1e3968cfc216b59bf16

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEeY3BEB897EKK3hJNaLi8sMUCOQUFAmZfIwgACgkQaLi8sMUC
OQXZYA/9ElVoUy3Um3IXFSwUGO+ctkvKd6idD7RuOBjqZyfadr4emrDrfQKYbCpa
Gik4M1H/GWobO/RaDjeSjQtGUmlPn8anhoFzmI6pPz7fBSfg5VGemllyHI2ypPpf
cJ1jLrmzpDGxLqPd/R/WsoE8dY9E7q20JgNESAqEYyjmjxqOjx6EnIjBjy8u+xL3
YWBw5BQn/1XbLXw4X7WJNH1cNIIZDgePdIb8Wq6wEDTzFzAvfw5BPhJ2rVaChV9P
6d25htXLy5FU/qvomiy1C+ZskzbZPKGDNgr8lC/MPeNgLi0d/ps2Rgut/CGjKreW
UiBmp3xslizR2/WhpRrcz0VLYxdNolfPY0odpgXkvQSEqGiZ1gOw5OQIN0f8HMiL
nOXnnxFVgdO/I/x9X2DwKAGwuts/GSeWOHdeNxvflyDGEYJHt9YMT7kXcJ0/dl6z
QSNHDoCMzMkxBCX23mlgY8pDSjw0Lqud0HDIChi1DFuNk7m1SfMIKGOn0ZAPsNqX
RuMiLCMOPzdE8BBBpKFwZFtx0zyC78xAOBK1M8DqlUexT3CBGFjOwCmGY27dLFZe
6ygdrqptb5uDOXFsw63cWSOilCnEcx7M8FDX7QjuV6EUQwvsxpeKvHZIFVlJNQCX
L5F8Lig/y4Q9iCjGiu3oT5zPuuEXPhKkyPsIeM9lC+zP/eC8rL4=
=E7lp
-----END PGP SIGNATURE-----

To view all past canaries, see:

• https://www.buskill.in/category/Canary/

What is BusKill?

BusKill is a laptop kill-cord. It's a USB cable with a magnetic breakaway that you attach to your body and connect to your computer.

Watch the BusKill Explainer Video for more info youtube.com/v/qPwyoD_cQR4

If the connection between you to your computer is severed, then your device will lock, shutdown, or shred its encryption keys -- thus keeping your encrypted data safe from thieves that steal your device.

Price hikes of over 2x widely expected under Broadcom’s VMware, survey finds

300 director-level IT workers making VMware decisions were questioned.

Hello friends. Work email is crushing me. The ticketing systems, plural, email me on everyone's tucket. (Because some people only work tickets via email and others through the web interface.)

Are there any email clients or servers that allow new email to land somewhere other than the inbox? Or allow my view to start elsewhere?

I declare email bankruptcy daily....

Send whiskey.

Edit: I was unclear.

I have filtering, but those all happen after the mail is in the Inbox. I get a quarter second of crazy emails and previews and things moving, then they are gone. (Outlook sucks.)

I don't even want to see that shit. Not at all.

Hello everyone, I hope I came to the right place to ask my stupid question. I'm currently working in a company that stuck way way back in time. I'm talking some people are still working in DOS level back in time. There is some revamping of this in progres, but it's going to be a long run.

The "IT department" (called computing department, lol) consist basically of 2 people, boss and me. Boss loves the old days, he's "happily" using Win XP on his computer and hates everything newer than Win7, although half of everything he tries to do doesn't work there anymore (and don't even start with security of the OS). Anyway... that's about the company background and what to expect.

During currently ongoing upgrades to get this company at least to 21st cwntury, there will be some sw licencing happening. I'm expecting like buying dozens of Office 2021 keys and some other standalone soft too. But there's problem with managing those keys, as there is no precedence, no rules and everything's going to be from the scratch almost. People are coming and going, PCs are dying (remember, there are still some DOS machines), hard drives replacing, etc. etc. Windows domain? Not in this company. Ever.

So, how do you keep track of what is installed and where? Thank you very much.

Hello!

My parents are moving to Italy and need a network. They only have 4G. I'm a little lost when it comes to UniFi and looking for some feedback if I've thought of everything and the setup works.

I was thinking Cloud Key 2, VPN Router Express, LTE Router UMR, maybe a U6 Lite AP to complement the one integrated into the Router if the range in the latter is insufficient, and a US-8-60W for power since only the cloud key and the AP would be powered using it. If they want cameras, I wanted to upgrade to CK2+, add G5 Bullets and use the US-8-150W instead.

I'm not sure whether I need the VPN router when the LTE router is already a router, too, but apparently the LTE router is not really integrated into the UniFi system. I need an AP, anyway, to the VPN Router is fine I think.

What do you think? Am I missing something? Thanks in advance.

---

Wait, is the UDR sufficient for the first setup? No cloud key, no vpn router, no switch...

So I have been a part of this community for a while and it seems pretty quiet. I know Lemmy is not as big as Reddit so this community will always be much smaller but I kind of miss the activity on r/sysadmin. Infinity for Reddit still works for view only so I have been scrolling though posts on Reddit as some of the stories and discussion there are fun to read.

With that being said, I think we can work to grow this community a bit. From what I can tell this community is home to a lot of quick posting. I am responsible somewhat as I have posted a bunch of articles. However, I am going to make a point to do longer write ups and I think it would be good we posted some stories. Additionally, I would be more than happy to help setup automatic posting for patch Tuesdays and similar scheduled posts.

As far as growth goes, I think we need to get the word out. A lot of people just do not know that Lemmy is a thing. If we can create some more meaningful posts and get some people to come over here from other platforms then I think this community will grow. I also know that mastodon is a pretty big platform so if we can get some people to engage from mastodon it will help as well.

I have created a new Samba community on lemmy.sdf.org

You can find it here:

[email protected]

My goal is to create a support network that is outside of the mailing list.