Harsh Weather Conditions (HWC) are environmental hazards that directly affect construction worker's health and safety, with a corresponding negative impact on their productivity. Climate predictions predict worsening scenarios of hot and humid weather conditions extending into regions not previously subject to HWC. The predictions pose a significant potential risk and elements of current practices in the construction industry seriously compromise workers' safety and productivity. HWC have been shown to have impacts on individuals and influence working environments which taken together lead to inhibiting the achievement of practical, safe and productive practices. This research aims to highlight the need to automatically identify the construction activity intensity in HWC, with proposing a conceptual framework of computerized vision analysis (CVA) system. This framework includes construction activity matrices that classify steel, formwork, and concreate work into macro and micro movements, with including the engaged materials, tools, and equipment. This system is expected to help in measuring the impacts of HWC on construction workers' safety and its implications on their productivity by identifying the activity intensity level and worker's idle time at the crew levels.