'Departments' such as the Department of Energy, the Department of Transportation, the State Department etc. are under the umbrella of the Executive Branch. Each Department has a Secretary. Secretary of Health and Human Services, Secretary of Defense, Secretary of Urban Development etc. The Secretaries act as liasons for their respective department and report directly to the Chief Executive (president). Given the president is Chief Executive he can give orders to the Departments which "they have to follow" provided they are not illegal or are beyond the department's budget. These issues, in principle, would be handled by the Judicial and Legislative branches respectively.
All that said, since the president is "in charge" of the Departments he can order them to do whatever he wants in addition. Typically, the president "tells" the Departments what he "wants" done and they will address that issue. An Executive Order is more forceful.
From a constitutional perspective, the problem with Executive Orders and Departments is that they aren't addressed at all. Constitution establishes the existance of the branches of goverment and lays out general rules, but it doesn't have anything to say about the structure of the executive branch or even how many judges are on the Supreme Court.
The only way executive orders can be addressed is by law. The legislative branch would need to write amd pass it with 2/3 majority vote to bypass a certain presidential veto.