Redis隨記--Lazy Free特性

Lazy Free特性

惰性刪除或延遲釋放(Lazy Free),指在刪除KEY時，采用異步方式延遲釋放EKY所使用的內存，將該操作交給單獨的子線程BIO(backgroup I/O)進行處理，避免在同步方式刪除KEY對Redis主線程的長期占用而影響系統可用性。

在刪除超大KEY如單個EKY占用內存過多或單個KEY包含過多元素時，同步刪除方式會導致Redis服務長期不可用，設置會引發自從主動故障切換。

在Redis 4.0版本開始提供惰性刪除特性。

Lazy Free使用場景

• 主動使用惰性刪除特性
• 被動使用惰性刪除特性

主動惰性刪除操作

• UNLINK命令

使用UNLINK刪除集合鍵時，會按照集合鍵的元素去估算釋放該KEY的成本，
如果釋放成本超過LAZYFREE_THRESHOLD，則會采用Lazy Free方式進行處理。

• FLUSHALL/FLUSHDB

通過ASYNC選項來設置FLUSHALL操作或FLUSHDB操作是否采用Lazy Free方式處理。

被動使用惰性刪除

被動使用惰性刪除主要有下面四類場景,並通過四個參數進行控制：

## 在內存到達最大內存需要逐出數據時使用
## 建議關閉，避免內存未及時釋放
lazyfree-lazy-eviction no

## 在KEY過期時使用
## 建議開啟
lazyfree-lazy-expire no

## 隱式刪除服務器數據時，如RENAME操作
## 建議開啟
lazyfree-lazy-server-del no

## 在對從庫進行全量數據同步時
## 建議關閉
slave-lazy-flush no

UNLINK命令涉及代碼

void delCommand(client *c) {
    delGenericCommand(c,0);
}

void unlinkCommand(client *c) {
    delGenericCommand(c,1);
}

/* This command implements DEL and LAZYDEL. */
void delGenericCommand(client *c, int lazy) {
    int numdel = 0, j;

    for (j = 1; j < c->argc; j++) {
        expireIfNeeded(c->db,c->argv[j]);
        int deleted  = lazy ? dbAsyncDelete(c->db,c->argv[j]) :
                              dbSyncDelete(c->db,c->argv[j]);
        if (deleted) {
            signalModifiedKey(c->db,c->argv[j]);
            notifyKeyspaceEvent(NOTIFY_GENERIC,
                "del",c->argv[j],c->db->id);
            server.dirty++;
            numdel++;
        }
    }
    addReplyLongLong(c,numdel);
}

/* Delete a key, value, and associated expiration entry if any, from the DB.
 * If there are enough allocations to free the value object may be put into
 * a lazy free list instead of being freed synchronously. The lazy free list
 * will be reclaimed in a different bio.c thread. */
#define LAZYFREE_THRESHOLD 64
int dbAsyncDelete(redisDb *db, robj *key) {
    /* Deleting an entry from the expires dict will not free the sds of
     * the key, because it is shared with the main dictionary. */
    if (dictSize(db->expires) > 0) dictDelete(db->expires,key->ptr);

    /* If the value is composed of a few allocations, to free in a lazy way
     * is actually just slower... So under a certain limit we just free
     * the object synchronously. */
    /* 進行Unlink操作，但不進行FREE操作 */
    dictEntry *de = dictUnlink(db->dict,key->ptr);
    if (de) {
        robj *val = dictGetVal(de);
        /* 計算lazy free當前對象的成本 */
        size_t free_effort = lazyfreeGetFreeEffort(val);

        /* If releasing the object is too much work, do it in the background
         * by adding the object to the lazy free list.
         * Note that if the object is shared, to reclaim it now it is not
         * possible. This rarely happens, however sometimes the implementation
         * of parts of the Redis core may call incrRefCount() to protect
         * objects, and then call dbDelete(). In this case we'll fall
         * through and reach the dictFreeUnlinkedEntry() call, that will be
         * equivalent to just calling decrRefCount(). */
        /* 當lazy free成本超過64時，使用后台線程處理 */
        if (free_effort > LAZYFREE_THRESHOLD && val->refcount == 1) {
            /* 對需要lazy free的對象數量+1 */
            atomicIncr(lazyfree_objects,1);
            /* 使用BIO子線程后台處理 */
            bioCreateBackgroundJob(BIO_LAZY_FREE,val,NULL,NULL);
            /* 將對象設置為NULL */
            dictSetVal(db->dict,de,NULL);
        }
    }

    /* Release the key-val pair, or just the key if we set the val
     * field to NULL in order to lazy free it later. */
    if (de) {
        dictFreeUnlinkedEntry(db->dict,de);
        if (server.cluster_enabled) slotToKeyDel(key);
        return 1;
    } else {
        return 0;
    }
}

/* Process the job accordingly to its type. */
if (type == BIO_CLOSE_FILE) {
	close((long)job->arg1);
} else if (type == BIO_AOF_FSYNC) {
	redis_fsync((long)job->arg1);
} else if (type == BIO_LAZY_FREE) {
	/* What we free changes depending on what arguments are set:
	 * arg1 -> free the object at pointer.
	 * arg2 & arg3 -> free two dictionaries (a Redis DB).
	 * only arg3 -> free the skiplist. */
	if (job->arg1)
		lazyfreeFreeObjectFromBioThread(job->arg1);
	else if (job->arg2 && job->arg3)
		lazyfreeFreeDatabaseFromBioThread(job->arg2,job->arg3);
	else if (job->arg3)
		lazyfreeFreeSlotsMapFromBioThread(job->arg3);
} else {
	serverPanic("Wrong job type in bioProcessBackgroundJobs().");
}

/* Release objects from the lazyfree thread. It's just decrRefCount()
 * updating the count of objects to release. */
void lazyfreeFreeObjectFromBioThread(robj *o) {
    decrRefCount(o);
    atomicDecr(lazyfree_objects,1);
}

void decrRefCount(robj *o) {
    if (o->refcount == 1) {
        switch(o->type) {
        case OBJ_STRING: freeStringObject(o); break;
        case OBJ_LIST: freeListObject(o); break;
        case OBJ_SET: freeSetObject(o); break;
        case OBJ_ZSET: freeZsetObject(o); break;
        case OBJ_HASH: freeHashObject(o); break;
        case OBJ_MODULE: freeModuleObject(o); break;
        case OBJ_STREAM: freeStreamObject(o); break;
        default: serverPanic("Unknown object type"); break;
        }
        zfree(o);
    } else {
        if (o->refcount <= 0) serverPanic("decrRefCount against refcount <= 0");
        if (o->refcount != OBJ_SHARED_REFCOUNT) o->refcount--;
    }
}

/* Free list object. */
void freeListObject(robj *o) {
    if (o->encoding == OBJ_ENCODING_QUICKLIST) {
        quicklistRelease(o->ptr);
    } else {
        serverPanic("Unknown list encoding type");
    }
}

/* Free entire quicklist. */
void quicklistRelease(quicklist *quicklist) {
    unsigned long len;
    quicklistNode *current, *next;

    current = quicklist->head;
    len = quicklist->len;
    while (len--) {
        next = current->next;

        zfree(current->zl);
        quicklist->count -= current->count;

        zfree(current);

        quicklist->len--;
        current = next;
    }
    zfree(quicklist);
}

UNLINK命令學習

• unlink命令和del命令都調用unlinkCommand函數，使用參數lazy來標識是否使用Lazy Free方式。
• Lazy Free方式會調用dbAsyncDelete函數來處理，處理過程中會計算Lazy Free方式釋放對象的成本，只有超過特定閾值，才會采用Lazy Free方式。
• Lazy Free方式會調用bioCreateBackgroundJob函數來使用BIO線程后台異步釋放對象。
• BIO線程后台處理時會采用DEL相同的方式來釋放對象，唯一區別是使用后台線程不會阻塞其他業務操作。
• 當Redis對象執行UNLINK操作后，對應的KEY會被立即刪除，不會被后續命令訪問到，對應的VALUE采用異步方式來清理。