Optimisation-based design is an established methodology that aims to achieve a globally optimal solution to a complex process design task by representing it as an optimisation problem. We propose a hybrid framework for decomposition-based process design, centred around hierarchical reinforcement learning and mathematical programming. The framework enables the agent to assemble processes, employ mathematical programming, and discover optimal designs without the need for a pre-defined process superstructure. The agent is composed of: (i) a higher level, that learns to construct the overall process by connecting process sections, and (ii) a lower level, that learns to build and solve sections by connecting and initialising unit operations. Such modularity allows for flexible and robust optimisation in constrained, nonlinear and nonconvex spaces. The framework is demonstrated in a case study of an intensified ethylene oxide production plant, yielding improved results compared to baseline designs reported in the open literature. The case study was implemented in Pyomo. Results reveal insights on the agent's learning speed and ability to leverage process models.