Featured image of post eBPF:使用eBPF捕获GnuTLS明文时,如何获取fd?

eBPF:使用eBPF捕获GnuTLS明文时,如何获取fd?

如何使用eBPF捕获GnuTLS的明文数据。

前情提要

当我们使用在eBPF跟踪那些使用libgnutls.so的加密传输时,不外乎是通过在gnutls_record_send()/gnutls_record_recv()这两个函数处部署uprobe以获取原始明文数据。而为了有序组织、聚合那些通过gnutls的原始数据,我们需要找出一个唯一的标识来作为聚合数据时的索引。gnutls应用于socket层之上,{pid,fd}能再好不过地成为那个索引。而本文的主题便是如何在eBPF使用捕获通过GnuTLS的明文时获取其关联的fd.

1
2
3
4
// https://man7.org/linux/man-pages/man3/gnutls_record_send.3.html
#include <gnutls/gnutls.h>
ssize_t gnutls_record_send(gnutls_session_t session, const void *
       data, size_t data_size);
1
2
3
4
// https://man7.org/linux/man-pages/man3/gnutls_record_recv.3.html
#include <gnutls/gnutls.h>
ssize_t gnutls_record_recv(gnutls_session_t session, void * data,
       size_t data_size);

send/recv分析

tips: 本节篇幅较长,可直接看结论.

首先肯定是需要下载GnuTLS进行分析的,可以以下链接下载源码:

gnutls / GnuTLS · GitLab

Index of /pub/net/gnupg/gnutls

相关数据结构

肯定是得从gnutls_record_send()/gnutls_record_recv() 入手分析,一眼可看到gnutls_session_t会话信息参数,该结构体定义于$gnutls_source/lib/gnutls_int.h.

1
2
3
4
5
6
struct gnutls_session_int {
    security_parameters_st security_parameters;
    record_parameters_st *record_parameters[MAX_EPOCH_INDEX];
    internals_st internals;
    gnutls_key_st key;
};

一般来说,fd肯定是隐藏在这个会话结构体中的,但是在往下查找里面的各个字段的结构时,就会发现几乎每个结构都挺庞大的,特别是internals_st这个字段.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
// gnutls3.6.16
typedef struct {
    /* holds all the parsed data received by the record layer */
    mbuffer_head_st record_buffer;

    int handshake_hash_buffer_prev_len;    /* keeps the length of handshake_hash_buffer, excluding
                         * the last received message */
    unsigned handshake_hash_buffer_client_hello_len; /* if non-zero it is the length of data until the client hello message */
    unsigned handshake_hash_buffer_client_kx_len;/* if non-zero it is the length of data until the
                         * the client key exchange message */
    unsigned handshake_hash_buffer_server_finished_len;/* if non-zero it is the length of data until the
                         * the server finished message */
    unsigned handshake_hash_buffer_client_finished_len;/* if non-zero it is the length of data until the
                         * the client finished message */
    gnutls_buffer_st handshake_hash_buffer;    /* used to keep the last received handshake
                         * message */

    bool resumable;    /* TRUE or FALSE - if we can resume that session */

    send_ticket_state_t ticket_state; /* used by gnutls_session_ticket_send() */
    bye_state_t bye_state; /* used by gnutls_bye() */
    reauth_state_t reauth_state; /* used by gnutls_reauth() */

    handshake_state_t handshake_final_state;
    handshake_state_t handshake_state;    /* holds
                         * a number which indicates where
                         * the handshake procedure has been
                         * interrupted. If it is 0 then
                         * no interruption has happened.
                         */

    bool invalid_connection;    /* true or FALSE - if this session is valid */

    bool may_not_read;    /* if it's 0 then we can read/write, otherwise it's forbidden to read/write
                 */
    bool may_not_write;
    bool read_eof;        /* non-zero if we have received a closure alert. */

    int last_alert;        /* last alert received */

    /* The last handshake messages sent or received.
     */
    int last_handshake_in;
    int last_handshake_out;

    /* priorities */
    struct gnutls_priority_st *priorities;

    /* variables directly set when setting the priorities above, or
     * when overriding them */
    bool allow_large_records;
    bool allow_small_records;
    bool no_etm;
    bool no_ext_master_secret;
    bool allow_key_usage_violation;
    bool allow_wrong_pms;
    bool dumbfw;

    /* old (deprecated) variable. This is used for both srp_prime_bits
     * and dh_prime_bits as they don't overlap */
    /* For SRP: minimum bits to allow for SRP
     * use gnutls_srp_set_prime_bits() to adjust it.
     */
    uint16_t dh_prime_bits; /* srp_prime_bits */

    /* resumed session */
    bool resumed;    /* RESUME_TRUE or FALSE - if we are resuming a session */

    /* server side: non-zero if resumption was requested by client
     * client side: non-zero if we set resumption parameters */
    bool resumption_requested;
    security_parameters_st resumed_security_parameters;
    gnutls_datum_t resumption_data; /* copy of input to gnutls_session_set_data() */

    /* These buffers are used in the handshake
     * protocol only. freed using _gnutls_handshake_io_buffer_clear();
     */
    mbuffer_head_st handshake_send_buffer;
    mbuffer_head_st handshake_header_recv_buffer;
    handshake_buffer_st handshake_recv_buffer[MAX_HANDSHAKE_MSGS];
    int handshake_recv_buffer_size;

    /* this buffer holds a record packet -mostly used for
     * non blocking IO.
     */
    mbuffer_head_st record_recv_buffer;    /* buffer holding the unparsed record that is currently
                         * being received */
    mbuffer_head_st record_send_buffer;    /* holds cached data
                         * for the gnutls_io_write_buffered()
                         * function.
                         */
    size_t record_send_buffer_user_size;    /* holds the
                         * size of the user specified data to
                         * send.
                         */

    mbuffer_head_st early_data_recv_buffer;
    gnutls_buffer_st early_data_presend_buffer;

    record_send_state_t rsend_state;
    /* buffer used temporarily during key update */
    gnutls_buffer_st record_key_update_buffer;
    gnutls_buffer_st record_presend_buffer;    /* holds cached data
                         * for the gnutls_record_send()
                         * function.
                         */

    /* buffer used temporarily during TLS1.3 reauthentication */
    gnutls_buffer_st reauth_buffer;

    time_t expire_time;    /* after expire_time seconds this session will expire */
    const struct mod_auth_st_int *auth_struct;    /* used in handshake packets and KX algorithms */

    /* this is the highest version available
     * to the peer. (advertized version).
     * This is obtained by the Handshake Client Hello
     * message. (some implementations read the Record version)
     */
    uint8_t adv_version_major;
    uint8_t adv_version_minor;

    /* if this is non zero a certificate request message
     * will be sent to the client. - only if the ciphersuite
     * supports it. In server side it contains GNUTLS_CERT_REQUIRE
     * or similar.
     */
    gnutls_certificate_request_t send_cert_req;

    size_t max_handshake_data_buffer_size;

    /* PUSH & PULL functions.
     */
    gnutls_pull_timeout_func pull_timeout_func;
    gnutls_pull_func pull_func;
    gnutls_push_func push_func;
    gnutls_vec_push_func vec_push_func;
    gnutls_errno_func errno_func;
    /* Holds the first argument of PUSH and PULL
     * functions;
     */
    gnutls_transport_ptr_t transport_recv_ptr;
    gnutls_transport_ptr_t transport_send_ptr;

    /* STORE & RETRIEVE functions. Only used if other
     * backend than gdbm is used.
     */
    gnutls_db_store_func db_store_func;
    gnutls_db_retr_func db_retrieve_func;
    gnutls_db_remove_func db_remove_func;
    void *db_ptr;

    /* post client hello callback (server side only)
     */
    gnutls_handshake_post_client_hello_func user_hello_func;
    /* handshake hook function */
    gnutls_handshake_hook_func h_hook;
    unsigned int h_type;    /* the hooked type */
    int16_t h_post;        /* whether post-generation/receive */

    gnutls_keylog_func keylog_func;

    /* holds the selected certificate and key.
     * use _gnutls_selected_certs_deinit() and _gnutls_selected_certs_set()
     * to change them.
     */
    gnutls_pcert_st *selected_cert_list;
    uint16_t selected_cert_list_length;
    struct gnutls_privkey_st *selected_key;

    /* new callbacks such as gnutls_certificate_retrieve_function3
     * set the selected_ocsp datum values. The older OCSP callback-based
     * functions, set the ocsp_func. The former takes precedence when
     * set.
     */
    gnutls_ocsp_data_st *selected_ocsp;
    uint16_t selected_ocsp_length;
    gnutls_status_request_ocsp_func selected_ocsp_func;
    void *selected_ocsp_func_ptr;
    bool selected_need_free;


    /* This holds the default version that our first
     * record packet will have. */
    uint8_t default_record_version[2];
    uint8_t default_hello_version[2];

    void *user_ptr;

    /* Holds 0 if the last called function was interrupted while
     * receiving, and non zero otherwise.
     */
    bool direction;

    /* If non zero the server will not advertise the CA's he
     * trusts (do not send an RDN sequence).
     */
    bool ignore_rdn_sequence;

    /* This is used to set an arbitrary version in the RSA
     * PMS secret. Can be used by clients to test whether the
     * server checks that version. (** only used in gnutls-cli-debug)
     */
    uint8_t rsa_pms_version[2];

    /* To avoid using global variables, and especially on Windows where
     * the application may use a different errno variable than GnuTLS,
     * it is possible to use gnutls_transport_set_errno to set a
     * session-specific errno variable in the user-replaceable push/pull
     * functions.  This value is used by the send/recv functions.  (The
     * strange name of this variable is because 'errno' is typically
     * #define'd.)
     */
    int errnum;

    /* A handshake process has been completed */
    bool initial_negotiation_completed;
    void *post_negotiation_lock; /* protects access to the variable above
                      * in the cases where negotiation is incomplete
                      * after gnutls_handshake() - early/false start */

    /* The type of transport protocol; stream or datagram */
    transport_t transport;

    /* DTLS session state */
    dtls_st dtls;
    /* Protect from infinite loops due to GNUTLS_E_LARGE_PACKET non-handling
     * or due to multiple alerts being received. */
    unsigned handshake_suspicious_loops;
    /* should be non-zero when a handshake is in progress */
    bool handshake_in_progress;

    /* if set it means that the master key was set using
     * gnutls_session_set_master() rather than being negotiated. */
    bool premaster_set;

    unsigned int cb_tls_unique_len;
    unsigned char cb_tls_unique[MAX_VERIFY_DATA_SIZE];

    /* starting time of current handshake */
    struct timespec handshake_start_time;

    /* expected end time of current handshake (start+timeout);
     * this is only filled if a handshake_time_ms is set. */
    struct timespec handshake_abs_timeout;

    /* An estimation of round-trip time under TLS1.3; populated in client side only */
    unsigned ertt;

    unsigned int handshake_timeout_ms;    /* timeout in milliseconds */
    unsigned int record_timeout_ms;    /* timeout in milliseconds */

    /* saved context of post handshake certificate request. In
     * client side is what we received in server's certificate request;
     * in server side is what we sent to client. */
    gnutls_datum_t post_handshake_cr_context;
    /* it is a copy of the handshake hash buffer if post handshake is used */
    gnutls_buffer_st post_handshake_hash_buffer;

/* When either of PSK or DHE-PSK is received */
#define HSK_PSK_KE_MODES_RECEIVED (HSK_PSK_KE_MODE_PSK|HSK_PSK_KE_MODE_DHE_PSK|HSK_PSK_KE_MODE_INVALID)

#define HSK_CRT_VRFY_EXPECTED 1
#define HSK_CRT_ASKED (1<<2)
#define HSK_HRR_SENT (1<<3)
#define HSK_HRR_RECEIVED (1<<4)
#define HSK_CRT_REQ_SENT (1<<5)
#define HSK_KEY_UPDATE_ASKED (1<<7) /* flag is not used during handshake */
#define HSK_FALSE_START_USED (1<<8) /* TLS1.2 only */
#define HSK_HAVE_FFDHE (1<<9) /* whether the peer has advertized at least an FFDHE group */
#define HSK_USED_FFDHE (1<<10) /* whether ffdhe was actually negotiated and used */
#define HSK_PSK_KE_MODES_SENT (1<<11)
#define HSK_PSK_KE_MODE_PSK (1<<12) /* client: whether PSK without DH is allowed,
                     * server: whether PSK without DH is selected. */
#define HSK_PSK_KE_MODE_INVALID (1<<13) /* server: no compatible PSK modes were seen */
#define HSK_PSK_KE_MODE_DHE_PSK (1<<14) /* server: whether PSK with DH is selected
                     * client: whether PSK with DH is allowed
                     */
#define HSK_PSK_SELECTED (1<<15) /* server: whether PSK was selected, either for resumption or not;
                  *        on resumption session->internals.resumed will be set as well.
                  * client: the same */
#define HSK_KEY_SHARE_SENT (1<<16) /* server: key share was sent to client */
#define HSK_KEY_SHARE_RECEIVED (1<<17) /* client: key share was received
                    * server: key share was received and accepted */
#define HSK_TLS13_TICKET_SENT (1<<18) /* client: sent a ticket under TLS1.3;
                     * server: a ticket was sent to client.
                     */
#define HSK_TLS12_TICKET_SENT (1<<19) /* client: sent a ticket under TLS1.2;
                       * server: a ticket was sent to client.
                       */
#define HSK_TICKET_RECEIVED (1<<20) /* client: a session ticket was received */
#define HSK_EARLY_START_USED (1<<21)
#define HSK_EARLY_DATA_IN_FLIGHT (1<<22) /* client: sent early_data extension in ClientHello
                      * server: early_data extension was seen in ClientHello
                      */
#define HSK_EARLY_DATA_ACCEPTED (1<<23) /* client: early_data extension was seen in EncryptedExtensions
                     * server: intend to process early data
                     */
#define HSK_RECORD_SIZE_LIMIT_NEGOTIATED (1<<24)
#define HSK_RECORD_SIZE_LIMIT_SENT (1<<25) /* record_size_limit extension was sent */
#define HSK_RECORD_SIZE_LIMIT_RECEIVED (1<<26) /* server: record_size_limit extension was seen but not accepted yet */
#define HSK_OCSP_REQUESTED (1<<27) /* server: client requested OCSP stapling */
#define HSK_CLIENT_OCSP_REQUESTED (1<<28) /* client: server requested OCSP stapling */
#define HSK_SERVER_HELLO_RECEIVED (1<<29) /* client: Server Hello message has been received */

    /* The hsk_flags are for use within the ongoing handshake;
     * they are reset to zero prior to handshake start by gnutls_handshake. */
    unsigned hsk_flags;
    struct timespec last_key_update;
    unsigned key_update_count;
    /* Read-only pointer to the full ClientHello message */
    gnutls_buffer_st full_client_hello;
    /* The offset at which extensions start in the ClientHello buffer */
    int extensions_offset;

    gnutls_buffer_st hb_local_data;
    gnutls_buffer_st hb_remote_data;
    struct timespec hb_ping_start;    /* timestamp: when first HeartBeat ping was sent */
    struct timespec hb_ping_sent;    /* timestamp: when last HeartBeat ping was sent */
    unsigned int hb_actual_retrans_timeout_ms;    /* current timeout, in milliseconds */
    unsigned int hb_retrans_timeout_ms;    /* the default timeout, in milliseconds */
    unsigned int hb_total_timeout_ms;    /* the total timeout, in milliseconds */

    bool ocsp_check_ok;    /* will be zero if the OCSP response TLS extension
                     * check failed (OCSP was old/unrelated or so). */

    heartbeat_state_t hb_state;    /* for ping */

    recv_state_t recv_state;    /* state of the receive function */

    /* if set, server and client random were set by the application */
    bool sc_random_set;

#define INT_FLAG_NO_TLS13 (1LL<<60)
    uint64_t flags; /* the flags in gnutls_init() and GNUTLS_INT_FLAGS */

    /* a verify callback to override the verify callback from the credentials
     * structure */
    gnutls_certificate_verify_function *verify_callback;
    gnutls_typed_vdata_st *vc_data;
    gnutls_typed_vdata_st vc_sdata;
    unsigned vc_elements;
    unsigned vc_status;
    unsigned int additional_verify_flags; /* may be set by priorities or the vc functions */

    /* we append the verify flags because these can be set,
     * either by this function or by gnutls_session_set_verify_cert().
     * However, we ensure that a single profile is set. */
#define ADD_PROFILE_VFLAGS(session, vflags) do { \
    if ((session->internals.additional_verify_flags & GNUTLS_VFLAGS_PROFILE_MASK) && \
        (vflags & GNUTLS_VFLAGS_PROFILE_MASK)) \
        session->internals.additional_verify_flags &= ~GNUTLS_VFLAGS_PROFILE_MASK; \
    session->internals.additional_verify_flags |= vflags; \
    } while(0)

    /* the SHA256 hash of the peer's certificate */
    uint8_t cert_hash[32];
    bool cert_hash_set;

    /* The saved username from PSK or SRP auth */
    char saved_username[MAX_USERNAME_SIZE+1];
    int saved_username_size;

    /* Needed for TCP Fast Open (TFO), set by gnutls_transport_set_fastopen() */
    tfo_st tfo;

    struct gnutls_supplemental_entry_st *rsup;
    unsigned rsup_size;

    struct hello_ext_entry_st *rexts;
    unsigned rexts_size;

    struct { /* ext_data[id] contains data for extension_t id */
        gnutls_ext_priv_data_t priv;
        gnutls_ext_priv_data_t resumed_priv;
        uint8_t set;
        uint8_t resumed_set;
    } ext_data[MAX_EXT_TYPES];

    /* In case of a client holds the extensions we sent to the peer;
     * otherwise the extensions we received from the client. This is
     * an OR of (1<<extensions_t values).
     */
    ext_track_t used_exts;

    gnutls_ext_flags_t ext_msg; /* accessed through _gnutls_ext_get/set_msg() */

    /* this is not the negotiated max_record_recv_size, but the actual maximum
     * receive size */
    unsigned max_recv_size;

    /* candidate groups to be selected for security params groups, they are
     * prioritized in isolation under TLS1.2 */
    const gnutls_group_entry_st *cand_ec_group;
    const gnutls_group_entry_st *cand_dh_group;
    /* used under TLS1.3+ */
    const gnutls_group_entry_st *cand_group;

    /* the ciphersuite received in HRR */
    uint8_t hrr_cs[2];

    /* this is only used under TLS1.2 or earlier */
    int session_ticket_renew;

    tls13_ticket_st tls13_ticket;

    /* the amount of early data received so far */
    uint32_t early_data_received;

    /* anti-replay measure for 0-RTT mode */
    gnutls_anti_replay_t anti_replay;

    /* Protects _gnutls_epoch_gc() from _gnutls_epoch_get(); these may be
     * called in parallel when false start is used and false start is used. */
    void *epoch_lock;

    /* If you add anything here, check _gnutls_handshake_internal_state_clear().
     */
} internals_st;

如此多的字段在不熟悉GnuTLS的情况看着就头疼(要是熟悉的话那还用得分析).

函数调用分析

tips: 或者可以直接使用doxygen+graphviz生成调用关系图分析.

因此,还是得从send/recv的实现入手分析,其实现在文件$gnutls_source/lib/record.c$gnutls_source/lib/record.c中:

1
2
3
4
5
6
ssize_t
gnutls_record_send(gnutls_session_t session, const void *data,
           size_t data_size)
{
    return gnutls_record_send2(session, data, data_size, 0, 0);
}

可以看到gnutls_record_send2()接着调用_gnutls_send_tlen_int()发送数据:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
ssize_t
gnutls_record_send2(gnutls_session_t session, const void *data,
            size_t data_size, size_t pad, unsigned flags)
{
    // 省略了一些代码
    switch(session->internals.rsend_state) {
        case RECORD_SEND_NORMAL:
            return _gnutls_send_tlen_int(session, GNUTLS_APPLICATION_DATA,
                             -1, EPOCH_WRITE_CURRENT, data,
                             data_size, pad, MBUFFER_FLUSH);
    // 省略了一些代码
    }
}

_gnutls_send_tlen_int()接着调用_gnutls_io_write_flush():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ssize_t
_gnutls_send_tlen_int(gnutls_session_t session, content_type_t type,
              gnutls_handshake_description_t htype,
              unsigned int epoch_rel, const void *_data,
              size_t data_size, size_t min_pad,
              unsigned int mflags)
{
    // 省略了一些代码
    /* Only encrypt if we don't have data to send
     * from the previous run. - probably interrupted.
     */
    if (mflags != 0
        && session->internals.record_send_buffer.byte_length > 0) {
        ret = _gnutls_io_write_flush(session);
        if (ret > 0)
            cipher_size = ret;
        else
            cipher_size = 0;

        retval = session->internals.record_send_buffer_user_size;
    } else {
    // 省略了一些代码
    }
    // 省略了一些代码
    return retval;
}

_gnutls_io_write_flush()接着调用_gnutls_writev():

1
2
3
4
5
6
7
8
// $gnutls_source/lib/buffers.c
ssize_t _gnutls_io_write_flush(gnutls_session_t session)
{
    // 省略了一些代码
    ret = _gnutls_writev(session, iovec, i, tosend);
    // 省略了一些代码
    return sent;
}

_gnutls_writev()接着调用_gnutls_writev_emu(),其实在这里已经可以看到我们找的fd,不过即然已经看到此处,不妨继续看看 gnutls_record_send的最终使用系统函数是什么:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
// $gnutls_source/lib/buffers.c
static ssize_t
_gnutls_writev(gnutls_session_t session, const giovec_t * giovec,
           unsigned giovec_cnt, unsigned total)
{
    // 省略了一些代码
    // 在此其实已经可以找到fd了
    gnutls_transport_ptr_t fd = session->internals.transport_send_ptr;
    // 省略了一些代码
    if (no_writev == 0) {
            i = session->internals.vec_push_func(fd, giovec, giovec_cnt);
        } else {
            i = _gnutls_writev_emu(session, fd, giovec, giovec_cnt, 1);

        }
    // 省略了一些代码
    i = _gnutls_writev_emu(session, fd, giovec, giovec_cnt, 0);
    // 省略了一些代码
    return i;
}

其实上面的函数不仅调用了_gnutls_writev_emu()还调用了session->internals.vec_push_func(fd, giovec, giovec_cnt),不过查看_gnutls_writev_emu()的实现可以发现其最终也是调用vec_push_func().

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
static ssize_t
_gnutls_writev_emu(gnutls_session_t session, gnutls_transport_ptr_t fd,
           const giovec_t * giovec, unsigned int giovec_cnt, unsigned vec)
{
    // 省略了一些代码
    ret = session->internals.vec_push_func(fd, &giovec[j], 1);
    // 省略了一些代码
    session->internals.push_func(fd, p,left);
    return ret;
}

然后找到internals中的gnutls_vec_push_func vec_push_func;字段,全局搜索gnutls_vec_push_func发现:

$gnutls_source/lib/includes/gnutls/gnutls.h的gnutls_transport_set_vec_push_function()用于设置vec_push_func,该函数实现在$gnutls_source/lib/system_override.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
void
gnutls_transport_set_vec_push_function(gnutls_session_t session,
                       gnutls_vec_push_func vec_func)
{
    session->internals.push_func = NULL;
    session->internals.vec_push_func = vec_func;
}

void
gnutls_transport_set_push_function(gnutls_session_t session,
                   gnutls_push_func push_func)
{
    session->internals.push_func = push_func;
    session->internals.vec_push_func = NULL;
}

全局搜索代码发现其在此处$gnutls_source/lib/system/fastopen.c被调用:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
void
gnutls_transport_set_fastopen(gnutls_session_t session,
                  int fd, struct sockaddr *connect_addr, socklen_t connect_addrlen,
                  unsigned int flags)
{
// 省略了以上代码
#ifdef MSG_NOSIGNAL
    if (session->internals.flags & GNUTLS_NO_SIGNAL)
        session->internals.tfo.flags |= MSG_NOSIGNAL;
#endif

#ifdef _WIN32
    gnutls_transport_set_vec_push_function(session, NULL);
    gnutls_transport_set_push_function(session, tfo_send);
#else
    gnutls_transport_set_vec_push_function(session, tfo_writev);
#endif
}

在非win32的情况下调用的是tfo_writev:

1
2
3
4
5
6
7
static ssize_t
tfo_writev(gnutls_transport_ptr_t ptr, const giovec_t * iovec, int iovec_cnt)
{
    // 省略以上代码
    return sendmsg(fd, &hdr, p->flags);
    // 省略以下代码
}

小结

因此结论是:

  • gnutls_record_send()/gnutls_record_recv()最终的系统调用是sendmsg/recvmsg.

  • session.internals.transport_send_ptrsession.internals.transport_recv_ptr即fd.

  • 不过上面的分析是针对3.6.16版本的,版本不同实现可能也不同?比如3.0.x的就是writev/send/recv.

在bpf中获取gnutls_session关联的fd

通过上一节内容我们知道int fd = (int)session.internals.transport_recv_ptr;。程序中我们可以这样获取fd:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
// ssize_t gnutls_record_send(gnutls_session_t session, const void *data,
//         size_t data_size);
SEC("uprobe/gnutls_record_recv")
int uprobe_entry_gnutls_record_recv(struct *pt_regs* ctx) {
    // gnutls_session pointer
    void* gnutls_session = (void*)PT_REGS_PAM1(ctx);

    void* internals_ptr = gnutls_session + internals_offset;

    // transport_recv_ptr is fd!
    int32_t fd = bpf_probe_read_user(&fd, sizeof(fd),
                    internals_ptr + transport_recv_ptr_offset);
    // ... ...
    return 0;
}

SEC("uretprobe/gnutls_record_recv")
int uprobe_ret_gnutls_record_recv(struct *pt_regs* ctx) {
    // ... 省略 ...
    return 0;
}

然后,上面的代码在跟踪gnutls_record_recv时并不能获取到fd,因为我们并不知道

internals_offsettransport_recv_ptr_offset,于是为了获取fd我们还需要知道上面两个offset的值是多少。

因此,现在的问题是:如何获取internalstransport_recv_ptr的偏移?

求值internals和transport_send_ptr的偏移

遗憾的是,GnuTLS对外提供的接口并没有暴露gnutls_session_t的细节,仅仅在gnutls/gnutls.h中声明了该结构,而真正的定义在$gnutls_source/lib/gnutls_int.h.

因此,我们需要下载GnuTLS的源码,尝试使用gnutls_int.h求出偏移值。不幸的是,似乎gnutls_int.h这个文件包含的头文件太多了.所以只好尝试编译一下源码了,以3.7.3为例.

编译nettle

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# 编译nettle
# gnutls verison => 3.7.3
# nettle version => 3.7

# https://ftp.gnu.org/gnu/nettle/nettle-3.5.tar.gz
wget "https://ftp.gnu.org/gnu/nettle/nettle-3.7.tar.gz"
tar -xvf nettle-3.7.tar.gz
cd nettle-3.7
./configure --prefix=/home/test/gnutls/output  --disable-openssl --enable-shared --enable-mini-gmp
make && make install
cd ..

编译gmp

1
2
3
4
5
6
7
# https://gmplib.org/download/gmp/gmp-6.0.0.tar.xz
wget "https://gmplib.org/download/gmp/gmp-$gmp_verison.tar.xz"
tar -xvf gmp-$gmp_verison.tar.xz
cd gmp-6.0.0
./configure --prefix=/home/test/gnutls/output 
make && make install
cd ..

最后是编译gnutls

1
2
3
4
5
6
7
8
wget "http://www.ring.gr.jp/pub/net/gnupg/gnutls/v3.7/gnutls-3.7.3.tar.xz"
tar -xvf gnutls-3.7.3.tar.xz
export LDFLAGS="$LDFLAGS -L/home/test/gnutls/output/lib"
export LDFLAGS="$LDFLAGS -L/home/test/gnutls/output/lib64"
PKG_CONFIG_PATH=/home/test/gnutls/output/lib64/pkgconfig ./configure --prefix=/home/test/gnutls/output  --with-included-libtasn1 --with-included-unistring --without-p11-kit
make
make install
cd ..

最后是编译获取offset的代码 :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cmake_minimum_required(VERSION 3.10)
project (get_gnutls_offset)
set(CMAKE_C_STANDARD 11)

add_definitions (-DHAVE_LIBNETTLE=1)
set(EXECUTABLE_OUTPUT_PATH ${CMAKE_BINARY_DIR})

set(gnutls_version "3.7.3" CACHE STRING "gnutls version, such as 3.7.3")

# nettle
string(REGEX MATCH "^[0-9]+\\.[0-9]+" nettle_version "${gnutls_version}")

include_directories(../${gnutls_version}/gnutls-${gnutls_version}/lib
                    ../${gnutls_version}/gnutls-${gnutls_version}/lib/minitasn1/
                    ../${gnutls_version}/gnutls-${gnutls_version}/
                    ../${gnutls_version}/gnutls-${gnutls_version}/gl
                    ../${gnutls_version}/output/include
                    ../${gnutls_version}/nettle-${nettle_version})

# 输出变量的值
message("gnutls_version: ${gnutls_version}")
message("nettle_version: ${nettle_version}")

add_executable(get_offset main.c)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
#include <stdio.h>
#include "gnutls/gnutls.h" // $gnutls_source/lib/includes/gnutls/gnutls.h
#include "config.h"  // $gnutls_source/config.h
#include "gnutls_int.h" // $gnutls_source/lib/gnutls_int.h

int main(int argc, char **argv)
{
  if(argc != 2) {
    printf("usage: ./gnutls_offset ${version} \n \
    ./gnutls_offset 3_4_10 \n");
    return -1;
  }

  printf("#define gnutls_%s_internals_offset 0x%lx \n", 
            argv[1], offsetof(struct gnutls_session_int, internals));
  printf("#define gnutls_%s_transport_recv_ptr_offset  0x%lx \n", 
            argv[1], offsetof(internals_st, transport_recv_ptr));
  printf("#define gnutls_%s_transport_send_ptr_offset  0x%lx \n\n", 
            argv[1], offsetof(internals_st, transport_send_ptr));

  return 0;
} 

现在,我们有面临另外一个问题,就是不同版本的GnuTLS中上述的两个偏移也许不是一致的?(待验证)。于是,我们需要求值GnuTLS的各个版本的offset.

1
2
3
#define gnutls_3_7_3_internals_offset 0x120 
#define gnutls_3_7_3_transport_recv_ptr_offset  0x4e0 
#define gnutls_3_7_3_transport_send_ptr_offset  0x4e8 

获取libgnutls.so的版本

查阅GnuTLS的开发文档可知gnutls_check_version可获得libgnutls.so的版本。

1
const char * gnutls_check_version (const char * req_version);
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
template <class T>
T* DLSymbolToFptr(void* handle, const char * symbol_name) {
  // The templated form compares nicely to c-style function pointer typedefs.
  // Example usage:
  // auto myFunction = DLSymbolToFptr<int (FooQux &, const Baz *)>( h, "somesymbol");
  T* fptr = reinterpret_cast<T*>(dlsym(handle, symbol_name));

  const char* dlsym_error = dlerror();
  if (dlsym_error) {
    return nullptr;
  }

  return fptr;
}

typedef  const char * (versionfn)(const char * req_version);

void get_gnutls_version(const char*libpath) {
    void *handle = dlopen(libpath, RTLD_LAZY);
    if(handle == nullptr) {
        printf("open %s fail \n", libpath);
        return ;
    }

    const char* version_fn_symbol = "gnutls_check_version";

    auto version_fn = DLSymbolToFptr<versionfn>(handle, version_fn_symbol);

    if(version_fn) {
        const char* version = version_fn(NULL);
        if(version != NULL) {
            printf("gnutls version: %s\n", version);
        }
    }
    dlclose(handle);
}

当入参为NULL时以字符串的形式返回libgnutls.so的版本,比如"3.7.3".

1
2
3
LIBGNUTLS_VERSION_MAJOR
LIBGNUTLS_VERSION_MINOR
LIBGNUTLS_VERSION_PATCH

最后,我们可以求出每个版本的偏移值,然后根据版本号确定偏移值,这样在bpf内核态中就可以拿到fd啦.

链接

Index of /gnu/nettle

Index of /pub/net/gnupg/gnutls/v3.0

Index of /download/gmp/

gnutls: GnuTLS API Reference Manual

gnutls · GitLab

Core TLS API (GnuTLS 3.8.0)

libgnutls的安装 - 宋海宾 - 博客园 (cnblogs.com)

Linux源码安装gnutls_gnutls 安装_HideInTime的博客-CSDN博客

哦吼是一首歌。
Built with Hugo
Theme Stack designed by Jimmy