[16:41:46.674] src/tools/ci/cores_backtrace.sh linux /tmp/cores [16:41:47.390] dumping /tmp/cores/postgres-3-32904.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [16:41:50.859] [New LWP 32904] [16:41:50.859] [New LWP 31931] [16:41:50.859] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [16:41:51.632] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [16:41:52.542] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [16:41:53.819] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [16:41:56.391] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [16:41:56.391] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [16:41:58.434] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [16:42:03.869] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [16:42:07.671] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [16:42:10.346] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [16:42:11.644] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [16:42:12.963] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [16:42:14.648] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [16:42:17.459] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [16:42:18.381] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [16:42:21.744] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [16:42:23.278] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [16:42:23.748] [Thread debugging using libthread_db enabled] [16:42:23.748] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [16:42:24.957] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [16:42:26.382] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [16:42:27.439] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [16:42:29.492] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [16:42:30.334] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [16:42:34.478] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [16:42:36.788] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [16:42:37.596] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [16:42:40.509] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [16:42:41.851] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [16:42:46.539] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [16:42:49.213] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [16:42:51.724] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [16:42:54.821] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [16:42:56.528] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [16:43:01.572] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [16:43:04.514] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [16:43:06.172] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [16:43:11.697] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [16:43:14.277] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [16:43:16.999] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [16:43:19.848] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [16:43:21.671] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [16:43:23.616] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [16:43:25.877] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [16:43:26.368] Core was generated by `postgres: standby2: startup waiting for 000000010000000000000006 '. [16:43:26.368] Program terminated with signal SIGQUIT, Quit. [16:43:26.368] #0 0x00007f7918687fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.396] [Current thread is 1 (LWP 32904)] [16:43:26.512] [16:43:26.512] Thread 2 (Thread 0x7f791601ea40 (LWP 31931)): [16:43:26.512] #0 0x00007f791874aa61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.512] No symbol table info available. [16:43:26.512] #1 0x00007f7918738e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.512] No symbol table info available. [16:43:26.512] #2 0x00007f7918739497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.512] No symbol table info available. [16:43:26.512] #3 0x00007f7918738cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.512] No symbol table info available. [16:43:26.512] #4 0x00007f7918694a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.512] No symbol table info available. [16:43:26.512] #5 0x00005563420d8ef6 in RestoreArchivedFile (path=path@entry=0x7ffe39613750 "pg_wal/00000", xlogfname=xlogfname@entry=0x7ffe39613ba0 "00000001", '0' , "6", recovername=recovername@entry=0x556342bb05bc "RECOVERYXLOG", expectedSize=16777216, cleanupEnabled=) at ../src/backend/access/transam/xlogarchive.c:177 [16:43:26.512] xlogpath = "pg_wal/RECOVERYXLOG", '\000' , "\066", '\000' , "\b\000\000\000\000\000\000\000\234ͺBcU\000\000x6a9\376\177\000\000\220ͺBcU\000\000\177g\241BcU\000\000\240;a9\376\177\000\000w\005\273BcU\000\000x6a9\376\177\000\000u\005\273BcU\000\000\030\351\001\026y\177\000\000\333q\241BcU", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [16:43:26.512] xlogRestoreCmd = 0x556343b8a658 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/040_standby_failover_slots_sync/data/t_040_standby_failover_slots_sync_publisher_data/archives/00000001", '0' , "6\" \"pg_wal/RECOVERYXLOG\"" [16:43:26.512] lastRestartPointFname = "00000001", '0' , "5\000q\241BcU", '\000' , "\001\000\000;\200\000\000\300\366\201\030y\177\000\000`\367\201\030y\177\000\000;\200\000\000\002", '\000' , "\020Mi\030y\177\000\000@/a9\376\177", '\000' , "WA~\030y\177\000\000OA~\030y\177\000\000X\246\270CcU", '\000' , "@", '\000' ... [16:43:26.512] rc = [16:43:26.512] stat_buf = {st_dev = 93884807985507, st_ino = 335544320, st_nlink = 140158077271392, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 9104222883162677760, st_size = 0, st_blksize = 93884826297944, st_blocks = 140729861091648, st_atim = {tv_sec = 140729861091232, tv_nsec = 140729861091360}, st_mtim = {tv_sec = 0, tv_nsec = 93884826297944}, st_ctim = {tv_sec = 140158077998231, tv_nsec = 0}, __glibc_reserved = {93884807997311, 140729861094320, 93884809661673}} [16:43:26.512] restartSegNo = [16:43:26.512] restartRedoPtr = 83886120 [16:43:26.512] restartTli = 1 [16:43:26.512] __func__ = "RestoreArchivedFile" [16:43:26.512] #6 0x00005563420e7b86 in XLogFileRead (segno=140729861094224, segno@entry=6, emode=emode@entry=13, tli=tli@entry=1, source=source@entry=XLOG_FROM_ARCHIVE, notfoundOk=notfoundOk@entry=true) at ../src/backend/access/transam/xlogrecovery.c:4209 [16:43:26.512] xlogfname = "00000001", '0' , "6\000\001\000\000\000\000\000\000 @a9\376\177\000\000#6\375\031y\177\000\000\001", '\000' [16:43:26.512] activitymsg = "waiting for 00000001", '0' , "6\000\177\000\000\177g\241BcU\000\000\260\067\306BcU\000\000\255\067\306BcU\000\000\270>a9\376\177\000\000\310\363\352BcU\000" [16:43:26.512] path = "pg_wal/00000\000\000\000\000", '\060' , "\066\000pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\066\000\000\000\000\000\000\000\000Sl\241B\000\000\000\062\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\234ͺBcU\000\000\b\000\000\000\000\000\000\000\031s\272BcU\000\000\330:a9\376\177\000\000\025s\272BcU\000\000\177g\241BcU\000\000!s\272BcU\000\000\031s\272BcU\000\000\330:a9\376\177\000\000\025s\272BcU\000\000\030\351\001\026y\177\000\000\035q\241BcU", '\000' ... [16:43:26.512] fd = [16:43:26.512] __func__ = "XLogFileRead" [16:43:26.512] #7 0x00005563420e7d76 in XLogFileReadAnyTLI (segno=6, emode=emode@entry=13, source=XLOG_FROM_ANY) at ../src/backend/access/transam/xlogrecovery.c:4337 [16:43:26.512] hent = [16:43:26.512] tli = 1 [16:43:26.512] cell__state = {l = , i = } [16:43:26.512] path = "\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "\277y\241BcU\000\000\000\000\000\000\000\000\000\000\002\243VCcU\000\000\200\243VCcU\000\000\240\037\000\000\377\377\002\000\330\066\306BcU\000\000\021\243VCcU\000\000\200\243VCcU\000\000\304=a9\376\177\000\000\347\066\306BcU\000\000\200\243VCcU\000\000\002\000\000\000\000\000\000\000|\201\241BcU\000\000\306=a9\376\177\000\000\304=a9\376\177", '\000' , "\030\000\000\000\060\000\000\000\240=a9\376\177\000\000U\202\241BcU\000\000", '\177' ... [16:43:26.512] cell = [16:43:26.512] fd = [16:43:26.512] tles = 0x556343b9ce08 [16:43:26.512] __func__ = "XLogFileReadAnyTLI" [16:43:26.512] #8 0x00005563420e8f72 in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=100683560, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=100683536, replayTLI=1, replayLSN=100683536, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3770 [16:43:26.512] oldSource = [16:43:26.512] startWalReceiver = false [16:43:26.512] last_fail_time = 764095116760361 [16:43:26.512] now = [16:43:26.512] streaming_reply_sent = true [16:43:26.512] __func__ = "WaitForWALToBecomeAvailable" [16:43:26.512] #9 0x00005563420e9aa7 in XLogPageRead (xlogreader=0x556343b37f80, targetPagePtr=100679680, reqLen=3880, targetRecPtr=100683536, readBuf=0x556343b86e28 "\024\321\006") at ../src/backend/access/transam/xlogrecovery.c:3351 [16:43:26.512] private = 0x556343b37f60 [16:43:26.512] emode = 15 [16:43:26.512] targetPageOff = 16384 [16:43:26.512] targetSegNo = 6 [16:43:26.512] r = [16:43:26.512] __func__ = "XLogPageRead" [16:43:26.512] #10 0x00005563420f1f02 in ReadPageInternal (state=state@entry=0x556343b37f80, pageptr=pageptr@entry=100679680, reqLen=reqLen@entry=3880) at ../src/backend/access/transam/xlogreader.c:1060 [16:43:26.512] readLen = [16:43:26.512] targetPageOff = 16384 [16:43:26.512] targetSegNo = 6 [16:43:26.512] hdr = [16:43:26.512] #11 0x00005563420f3b5d in XLogDecodeNextRecord (state=state@entry=0x556343b37f80, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [16:43:26.512] RecPtr = 100683536 [16:43:26.512] record = [16:43:26.512] targetPagePtr = 100679680 [16:43:26.512] randAccess = false [16:43:26.512] len = [16:43:26.512] total_len = [16:43:26.512] targetRecOff = 3856 [16:43:26.512] pageHeaderSize = [16:43:26.512] assembled = false [16:43:26.512] gotheader = [16:43:26.512] readOff = [16:43:26.512] decoded = 0x0 [16:43:26.512] errormsg = 0x556342c51c76 "LOG" [16:43:26.512] #12 0x00005563420f46ff in XLogReadAhead (state=0x556343b37f80, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [16:43:26.512] result = [16:43:26.512] #13 0x00005563420df041 in XLogPrefetcherNextBlock (pgsr_private=93884825963952, lsn=0x556343b9c7a8) at ../src/backend/access/transam/xlogprefetcher.c:492 [16:43:26.512] nonblocking = [16:43:26.512] record = [16:43:26.512] prefetcher = 0x556343b38db0 [16:43:26.512] reader = 0x556343b37f80 [16:43:26.512] replaying_lsn = 100683488 [16:43:26.512] __func__ = "XLogPrefetcherNextBlock" [16:43:26.512] #14 0x00005563420e340a in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [16:43:26.512] No locals. [16:43:26.512] #15 lrq_complete_lsn (lsn=0, lrq=0x556343b9c6d8) at ../src/backend/access/transam/xlogprefetcher.c:290 [16:43:26.512] No locals. [16:43:26.512] #16 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x556343b38db0, errmsg=errmsg@entry=0x7ffe39618338) at ../src/backend/access/transam/xlogprefetcher.c:1037 [16:43:26.512] record = [16:43:26.512] replayed_up_to = 0 [16:43:26.512] #17 0x00005563420ea2d4 in ReadRecord (xlogprefetcher=0x556343b38db0, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3151 [16:43:26.512] errormsg = 0x556343b38790 "" [16:43:26.512] record = [16:43:26.512] xlogreader = 0x556343b37f80 [16:43:26.512] private = [16:43:26.512] __func__ = "ReadRecord" [16:43:26.512] #18 0x00005563420ed46b in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [16:43:26.512] xtime = [16:43:26.512] ru0 = {tv = {tv_sec = 1710779912, tv_usec = 197330}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1093}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 15760, __ru_maxrss_word = 15760}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 401, __ru_minflt_word = 401}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 4, __ru_nvcsw_word = 4}, {ru_nivcsw = 0, __ru_nivcsw_word = 0}}} [16:43:26.531] record = 0x7f79158d8f70 [16:43:26.531] reachedRecoveryTarget = false [16:43:26.531] replayTLI = 1 [16:43:26.531] __func__ = "PerformWalRecovery" [16:43:26.531] #19 0x00005563420d7118 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [16:43:26.531] Insert = [16:43:26.531] checkPoint = {redo = 83886120, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 749}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1710779911, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 749} [16:43:26.531] wasShutdown = false [16:43:26.531] didCrash = [16:43:26.531] haveTblspcMap = false [16:43:26.531] haveBackupLabel = true [16:43:26.531] EndOfLog = [16:43:26.531] EndOfLogTLI = [16:43:26.531] newTLI = [16:43:26.531] performedWalRecovery = [16:43:26.531] endOfRecoveryInfo = [16:43:26.531] abortedRecPtr = [16:43:26.531] missingContrecPtr = [16:43:26.531] oldestActiveXID = [16:43:26.531] promoted = false [16:43:26.531] __func__ = "StartupXLOG" [16:43:26.531] #20 0x000055634258811b in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [16:43:26.531] No locals. [16:43:26.531] #21 0x000055634257f8e5 in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:267 [16:43:26.531] pid = [16:43:26.531] #22 0x00005563425817b0 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [16:43:26.531] pid = [16:43:26.531] __func__ = "StartChildProcess" [16:43:26.531] #23 0x000055634258787a in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x556343b36e40) at ../src/backend/postmaster/postmaster.c:1364 [16:43:26.531] opt = [16:43:26.531] status = [16:43:26.531] userDoption = [16:43:26.531] listen_addr_saved = false [16:43:26.531] output_config_variable = [16:43:26.531] __func__ = "PostmasterMain" [16:43:26.531] #24 0x00005563423b31db in main (argc=4, argv=0x556343b36e40) at ../src/backend/main/main.c:197 [16:43:26.531] do_check_root = [16:43:26.531] [16:43:26.531] Thread 1 (LWP 32904): [16:43:26.531] #0 0x00007f7918687fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.531] No symbol table info available. [16:43:26.531] #1 0x00007f791873913a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.531] No symbol table info available. [16:43:26.531] #2 0x00007f791874aa6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [16:43:26.531] No symbol table info available.