Programming

As the other commenter said, you want to learn about programming principles. Like, low coupling or don't repeat yourself.

How long is your longest program? What would you say is a typical length?

You say your code is "bad" -- in what ways? For example:

• Readability (e.g. going back to it months later so you go "oh I remember" or "wtf does this do?!"
• Maintainability (go back to update and you have to totally rework a bunch of stuff for a change that seems like it should be simple)
• Reliability (mistakes, haphazard "testing", can't trust output)
• Maybe something else?

Learning new programming languages is an awesome way to expand your programming brain. If you want to stay in the same scientific computation niche, you can check out Julia or Mathematica. If you’re just looking to broaden your horizons, the world is your oyster. For me, learning Clojure really cooked my noodle but made me a much better programmer since it taught me functional programming.

Also, just read other peoples code! You can learn the conventions that way. Though for you it would best to find other products within your niche, because I’m not sure if general web dev code would be super helpful.

There are techniques that are broader than any single language’s conventions, and I think learning those are how you can improve. That’s hard to teach, though, and it comes from experience with a few different languages, in my opinion.

And honestly, I can totally respect the “conventions be damned” attitude, because at the end of the day, you’re trying to make something that works, and if nobody else is reading that code, you’ve made the right trade-off.

As one physicist to another, the most important thing in the code are long variable names (descriptive) and comments.

We usually do not do multi-people multi year projects, so all other comments in this page especially the ones coming from programmers are not that relevant. Classes are cool, but they are not needed and often obscure clarity of algorithmic/functional programming.

S. Wolfram (creator of Mathematica) said something along these lines (paraphrasing) if you are writing real code in Mathematica - you are doing something wrong.

As one physicist to another, the most important thing in the code are long variable names (descriptive) and comments.

We usually do not do multi-people multi year projects, so all other comments in this page especially the ones coming from programmers are not that relevant. Classes are cool, but they are not needed and often obscure clarity of algorithmic/functional programming.

S. Wolfram (creator of Mathematica) said something along these lines (paraphrasing) if you are writing real code in Mathematica - you are doing something wrong.

Learn Haskell.

Since it is a research language, it is packed with academically-rigorous implementations of advanced features (currying, lambda expressions, pattern matching, list comprehension, type classes/type polymorphism, monads, laziness, strong typing, algebraic data types, parser combinators that allow you to implement a DSL in 20 lines, making illegal states unrepresentable, etc) that eventually make their way into other languages. It will force you to learn some of the more advanced concepts in programming while also giving you a new perspective that will improve your code in any language you might use.

I was big into embedded C programming years back ... and when I got to the pointers part, I couldn't figure out why I suddenly felt unsatisfied and that I was somehow doing something wrong. That instinct ended up being at least partially correct. I sensed that I was doing something unsafe (which forced me to be very careful around footguns like pointers, dedicating extra mental processes to keep track of those inherently unsafe solutions) and I wished there was some more elegant way around unsafe actions like that (or at least some language provided way of making sure those unintended side effects could be enforced by the compiler, which would prevent these kinds of bugs from getting into my code).

Years later, after not enjoying JS, TS (IMO, a porous condom over the tip of JavaScript), Swift, Python, and others, my journey brought me to FRP which eventually brought me to FP and with it, Haskell, Purescript, Rust, and Nix. I now regularly feel the same satisfaction using those languages that I felt when solving a math problem correctly. Refactoring is a pleasure with strictly typed languages like that because the compiler catches almost everything before it will even let you compile.

This is only tangentially related to improving your code directly as you have asked. However, in a similar vein as using source control (git), when using Python learn to manage your environments. Venv, poetry, conda/mamba, etc are tools to look into.

I used to work with mostly scientists, and a good number of them knew some Python, but none of them knew how to properly manage their environments and it was a huge problem. They would often come to me and say "I ran this script a week ago and it worked, I tried it today without making any changes and it's throwing this error now that I don't understand." Every time it was because they accidentally changed their dependencies, using their global python install. It also made it a nightmare to try to revive old code for them, since there was almost no way to know what version of various libraries were used.

Do you want to work as a developer? Or do you want to want to continue with your research and analysis? If you're only writing code for your own purposes, I don't know why it matters if it's conventional.

Two books that may be helpful:

• Fluent Python by Luciano Ramalho
• Python Distilled by David M. Beazley

I'm more familiar with the former, and think it's very good, but it may not give you the basic introduction to object oriented programming (classes and all that) you're looking for; the latter should.

Think two things:

1. optimize the control flow of your code

2. make it easy to read

You should also be disciplined with these two ideas, your code will look better as you become more experienced, 100% guaranteed.

Could be good to try to 'reset' your brain, by learning an entirely new programming language. Ideally, a statically typed, strict language like Rust or Java, or Scala, if you happen to have a use for it in data processing. They'll partially force you to do it the proper way, which can be eye-opening and will translate backwards to Python et al.
Just in general, getting presented the condensate of a different approach to programming, by learning a new language, can teach a lot about programming, even if you're never going back to that language.

For learning more about Git, I can recommend Oh My Git!. It takes a few hours to get through. In my experience, it's really useful to have at least seen all the tools Git provides, because if something goes sideways, you can remedy it with that.

While there are lots of programming courses out there, not many of them will explicitly teach you about good programming principles. Here are a couple things off the top of my head:

• High cohesion, low coupling. That is, when you divide up code into functions and classes, try to minimize the number of things going between those functions (if your functions regularly have 6+ arguments, that's a red flag and should be reviewed). And when something needs to be broken up into pieces, try to find the spots where there are minimal points of contact.

• Try to divide code between functions and files in a way that doesn't feel too busy. If there are a bunch of related functions that are cluttering up one file, or that are referenced from multiple places, consider making a module for those. If you're not sure what "too busy" means...

• Read a style guide. There are lots of things that will help you clean up and organize your code. The guide won't necessarily tell you why to do each thing, but it's a great tool when you don't have another point of reference.

If you have a chance to take a "Software Engineering 101" class, this is where you'd learn most of the basic principles for writing better code.

Forget everything you hear about OOP and just view it as a way to improve code readability. Just rewrite something convoluted with a class and you'll se what they're good for. Once you've got over the mental blockade, it'll all make more sense.

Read your own code that you wrote a month ago. For every wtf moment, try to rewrite it in a clearer way. With time you will internalize what is or is not a good idea. Usually this means naming your constants, moving code inside function to have a friendly name that explain what this code does, or moving code out of a function because the abstraction you choose was not a good one. Since you have 10 years of experience it's highly possible that you already do that, so just continue :)

If you are motivated I would advice to take a look to Rust. The goal is not really to be able to use it (even if it's nice to be able able to write fast code to speed up your python), but the Rust compiler is like a very exigeant teacher that will not forgive any mistakes while explaining why it's not a good idea to do that and what you should do instead. The quality of the errors are crutial, this is what will help you to undertand and improve over time. So consider Rust as an exercice to become a better python programmer. So whatever you try to do in Rust, try to understand how it applies to python. There are many tutorials online. The official book is a good start. And in general learning new languages with a very different paradigm is the best way to improve since it will help you to see stuff from a new angle.

Check Udemy for courses and wait for a sale. They normally list for hundreds of dollars but routinely (pretty much monthly) for about $10 - $15 dollars.

You can learn the basics using any beginner's course on udemy.

Then I'd recommend that you build smaller projects to practice. In the future you can change the types of projects you're building to practice even more (rest apis, website frontend, messaging app, low level stuff with C/rust/golang), if you want yo reach this level, of course.

Approach programming with the same seriousness that you’d expect a programmer to approach your field with. You say yourself you just want it to “do the thing, conventions be damned”.

Well how would you feel if someone entered your lab or whatever and treated the tools of your trade that way?