Ask Lemmy

There are some pitfalls to be aware of that may not be very intuitive for someone who is not a scientist and even tricky if you are one:

1. the place where something is published matters and it can be hard to tell what is good and what is bad. If you work in a certain field for a few years and talk with your peers, you will get an idea how to read certain types of articles, depending on where they are published. Each field has their top journals/conferences and lower quality ones. If you conduct an amazing new experiment, you will try to get it published in the better ones. This doesn't mean that the other ones are complete crap, but there may be some problems with the research that you as an outsider won't see. The problem is, they are all called something like "International Top Conference/Journal for A Field With A Cool Sounding Name".

There were some embarrassing cases during the Covid pandemic where professors from different fields like economics tried to pose as virus experts because they also know statistics. So they tried to give critical comments about the virologists. But if you have never been in an actual lab where people work with viruses, you have no clue whether things like reasons for excluding certain cases from an analysis are legitimate. You also don't know which key variables you need to know (e.g., is temperature important for vaccine effectiveness? I don't know, but if it is, a virologist can tell you and an economist can't).

A proxy measure for this quality of conference/journal is the number of people who have cited an article. But this doesn't always help and can also be misleading, and some fields in the social sciences and humanities don't care about this at all. And even if it counts, it strongly varies by field. For example, medicine has really high citation counts (thus many of the top journals across disciplines) and mathematics has really low citation counts.

2. don't rely on only a single study. If you look for the light therapy example, one study is better than no study, but usually it helps if you have the time to read a few more studies. Even if one study finds an effect, it is not uncommon that was just due to pure randomness or bad practices during data analysis ("p hacking", "HARKing" etc. This is the best pathway, but very time intense. Even many scientists fail to read their literature properly to stay up to date (because you have tons of other stuff to do as well and the reality is that writing, not reading, keeps you in your job).

3. if you don't have a lot of time to read 10-20 articles, you might still be lucky and find a summary article about the topic. They are sometimes called "literature synthesis", "literature review", "systematic review", or "meta analysis" (good search terms, btw). If you find one that was published in a good journal/conference (or has let's say more than 100 citations if it was published at least 5 years ago - again, take this with a grain of salt), chances are high that's the gold nugget you are looking for. Read this thing properly and you either have a good overview or at least found more interesting studies to read.

Btw: If you can't download an article from for example google scholar, there are search engines where you can get almost anything for free (a good one is maintained by Alexandra Elbakyan). If that doesn't help, write to the authors directly. If it's a field of practical relevance, maybe you can even include the exact question you have and they may share their expertise and a few more sources with you.