Accurate diagnosis and segmentation of skin lesion is critical for early detection and diagnosis of skin cancer. Recent multi-task learning methods require expensive annotations for skin lesion analysis while single-task driven models cannot fully utilize the potential knowledge. The aim of this study is to utilize the neglected knowledge by a flexible architecture in dermoscopy skin lesion classification and segmentation. In this work, we propose a cascade knowledge diffusion network (CKDNet) to transfer and aggregate knowledge learnt from different tasks to simultaneously boost the performances of classification and segmentation. CKDNet consists of a sequence of coarse-level segmentation, classification, and fine-level segmentation networks. We design two novel feature entanglement modules, Entangle-Cls and Entangle-Seg, for classification and segmentation. The Entangle-Cls module aggregates the diffused features from initial segmentation to drive the classification network’s attention to image regions relevant to the disease. The Entangle-Seg module integrates the cascaded context knowledge learnt from classification to benefit fine-level segmentation, especially at uncertain boundaries. The entanglement modules can adaptively control the knowledge that can be diffused from one task to another, which avoids the empirical selection of weights for different learning tasks compared to other multi-task methods. We perform extensive evaluations and comparisons with state-of-the-art methods on skin lesion classification and segmentation with challenge datasets, ISIC2017 and ISIC2018. Our CKDNet demonstrated superior performance without using any ensemble approaches or any external datasets. The effectiveness of each component and loss functions are demonstrated by interpretable results using class activation maps (CAM), t-SNE, and classification and segmentation results.